Wednesday, 31 March 2021

Crx2Oak Migration tool Concepts and Demo

 Crx2Oak helps migrate data from older CQ versions based on Apache Jackrabbit 2 to Oak, and it can also be used to copy data between Oak repositories.


 

The tool can be used for:

  • Migrating from older CQ 5 versions to AEM 6
  • Copying data between multiple Oak repositories
  • Converting data between different Oak MicroKernel implementations. (S3DataStore to FileDataStore)


Some of its features are

  • The migration can be interrupted at any time, with the possibility to resume it afterwards.
  • Custom Java logic can also be implemented using CommitHooks.
  • CRX2Oak also supports memory mapped operations by default. Memory mapping greatly improves performance


Parameters
Node Store Options
--cache: Cache size in MB (default is 256)

--mmap: Enable memory mapped file access for Segment Store

--src-password: Password for the source RDB database

--src-user: User for the source RDB

--user: User for the targed RDB

--password: Password for the target RDB.

Version Store Options
--copy-orphaned-versions: Skips copying orphaned versions. Parameters supported are: true, false and yyyy-mm-dd. Defaults to true.

--copy-versions: Copies the version storage. Parameters: true, false, yyyy-mm-dd. Defaults to true.


Path Options
--include-paths: Comma-separated list of paths to include during copy
--merge-paths: Comma-separated list of paths to merge during copy
--exclude-paths: Comma-separated list of paths to exclude during copy.


Source Blob Store Options
--src-datastore: The datastore directory to be used as a source FileDataStore

--src-fileblobstore: The datastore directory to be used as a source FileBlobStore

--src-s3datastore: The datastore directory to be used for the source S3DataStore

--src-s3config: The configuration file for the source S3DataStore.


Destination BlobStore Options
--datastore: The datastore directory to be used as a target FileDataStore

--fileblobstore: The datastore directory to be used as a target FileBlobStore

--s3datastore: The datastore directory to be used for the target S3DataStore

--s3config: The configuration file for the target S3DataStore.

Help Options
-?, -h, --help: Shows help information.

Debugging
--log-level TRACE or --log-level DEBUG 

Demo Video Series

What are the option for asset migration between AEM instances?


There are cases where we need to move assets across AEM instances. This may occur when we have multiple asset repositories and as part of merging assets we may have to move them into one single instance. While moving assets we have to consider various factors like asset metadata, asset versions, Tags w.r.t assets all moved and validated.

The overall size of asset is another consideration while selecting any tools to migrate assets. Below given few tools which can be included in our consideration while we go for asset migration.


-Replication Agent
By configuring replication agent, we can migrate assets across AEM Instances

-Vault Remote Copy

Jackrabbit vault offers a simple method to copy nodes between repositories.

This can be used for bulk assets.

References:
https://jackrabbit.apache.org/filevault/rcp.html
https://experienceleague.adobe.com/docs/experience-manager-65/assets/administer/assets-migration-guide.html?lang=en#migrating-between-aem-instances
License: Jackrabbit Oak and any of its parts are licensed according to the terms listed in the Apache License, Version 2.0

-Grabbit

Grabbit provides a fast and reliable way of copying content from one Sling (specifically Adobe AEM) instance to another.
Grabbit creates a stream of data using Google’s Protocol Buffers aka "ProtoBuf". Protocol Buffers are an extremely efficient (in terms of CPU, memory and wire size) binary protocol that includes compression.

This is one of the Adobe recommended solution & can be considered for bulk assets movement.

Ref:
https://github.com/TWCable/grabbit
https://relevancelab.com/2019/07/04/get-moving-with-aem-and-grabbit/
License: Licensed under the Apache License, Version 2.0 https://github.com/TWCable/grabbit/blob/master/docs/LicenseInfo.adoc


-Recap

Crx sync option based on vlt rcp

Ref:
http://adamcin.net/net.adamcin.recap/

-Crx2Oak

Crx sync option  - tools provided by Adobe while upgrading AEM or for migration of crx data.
This is one of the Adobe recommended & can be used for bulk Assets

-S3 Asset Ingestor

This is part of ACS AEM Commons .It pulls files from an Amazon S3 bucket instead of the local file system. You can load a directory of assets into AEM very easily with this tool. Because of the ability to overload a server with assets, this tool only appears for the “admin” user right now.

Ref:
https://adobe-consulting-services.github.io/acs-aem-commons/features/mcp-tools/asset-ingestion/s3-asset-ingestor/index.html


What is AEP? Adobe Experience Platform FAQ



What is AEP(Adobe Experience Platform)
AEP helps to capture the customer journey. In AEP, data from various sources are stitched together using a schema. Thus an identity graph is built which is unique to a customer. AEP has a data lake where data from various sources are streamlined and fed into. This will be used to create profile data for a customer.




AEP is basically a combination of Real time customer profile + AI & machine Learning + Open Ecosystem

What are all the various data sets defined in AEP?
AEP has various data sets like,
Attributes: Characters like customer name, email, gender etc.
Identities: Unique identity info like ECID(experience cloud id), Email, membership id, phone no etc.
Segments : Categories like  online shoppers, gender, Location. [one such use case is, these segments can be exported to utilize in an email campaign]
Behaviors: Like login to the website, installed appication, added an item to cart etc.

What AEP Solves?
AEP solves below concerns.

  1. Disconnected identities.
  2. Slow and vulnerable data transfer.
  3. AI & ML operates in silos usually. Extraction of data is tough in such cases.
  4. Data governance is not enforced usually(CCPA, GDPR etc)
  5. Centralization of multiple features.


Capabilities or major AEP Features:

  • Create Actionable, real time intelligent customer profiles.
  • Enrich data and derive more insights with AI & ML & Data queries(SQL).
  • Innovate with open & composable components (Open APIS etc.)
  • Enhance delivery
  • Privacy and data protection(Privacy framework, consent offering, security )


AEP integration into other Adobe Cloud Applications
Adobe Experince cloud applications (Marketing cloud, Analytics cloud, Advertising cloud, commerce cloud) can be easily integrated to  AEP.

All customer attributes are fed into AEP from different applications.
For eg.
Adobe Analytics send data(when ever a data point is captured, immediately it goes into AEP), Adobe Target send data(decision made, content presented kind of data), Audience (send trace and audience), Campaign(profile and event data) can be easily fed into AEP via launch.

AEP uses Launch & websdk to import data directly into AEP from various applications.

How AEM or Forms utilizes AEP?
AEM can use this AEP data to personalize content on pages or forms.

Various AEP Implementation Phases & Roles responsible for.

Plan - (Leads and an enterprise architect will plan referring business goals and document it by defining KPIs)
Implement - (Data architects and engineers create data lakes(create model, schemas) and make available), ensure data integrity by query services.
Use - Marketrs, data analysts, data scientists(uses query service), application developer interacts with UI & start working towards integrating with other adobe applications(campaign,target, analytics)
Grow - People highlights the growth to the initial set of team (Enterprise architects, company lead etc)

Basic architecture of AEP

Through data ingestion (either third party ETL, ERP, Sales or Adobe applications via launch), we can ingest data into AEP data lake. The data resides as batches and files. Any data getting pushed also placed in Experience platform pipeline. Any data gets into AEP traverse to identity graph and profile store quickly.

The controls native to AEP are
Access control: specific permission rights to data & users
Data governance: to ensure data integrity
Experience data model systems: common data model, which cn be extended based on needs
Query service: SQL way of accessing data - it has connectors, so other sql tools can connect to this query service
Data science workspace: allows data scientists to create data models build train and deploy.
Intelligent services: like Attribution AI or customer AI - prebuild models can be configured to operate on data
Segmentation capabilities: for categorization. it includes streams and batches

Application Services
-Customer journey analytics - Combine all data from every channel. It has analysis workspace/ interface on top of AEP, helps visualize and explore all data from data lake.
-Real time customer data platform(CDP) - rich real time customer profiles, actionable insights, data governance. CDP has segmentation capabilities.
-Journey orchestration -  enables orchestration of triggered interactions like registration confirmations, location based information
-Intelligent Services -  Utilizing the AI & ML Capabilities to intelligently predict customer behavior.
-Offer decisioning - Build offer, apply decisioning & then deploy the right offer.

Use cases of AEP

1) Real time customer data platform - Stitch known and unknown data to activate customer profiles
2) Customer journey intelligence - Utilize the data driven methodologies, best practices AI & ML to enable real time decisions and actions
3) Delivery and cross channel experience - capability to deliver consistent and personalized experiences across all channels with the combination of platform and experience cloud products.
4) Customer experience application development - AEP gives an open and extensible platform for low latency access to profiles decisions and insights to create new customer experience applications.


Saturday, 2 January 2021

All you need to know about Adobe Client Data Layer

What is Adobe Client Data Layer

A data layer in general consists of a JavaScript client side event driven data store that can be used on webpages, to collect data about what the visitors experience on the web page & to communicate this data to digital analytics and reporting servers(e.g. Adobe analytics or Adobe target)

What does it do?

The Adobe client data layer is a java script store for data and events happening on a page within the scope of a request. It reduce the effort to instrument websites by providing a standardized method to expose and access any kind of data for any script. 

It provides API to,

1) Register data that is to be merged into the data layer state
2) Trigger events that relate to the data stored in the data layer
3) Get the current data layer state of all merged data
4) Register listeners that are called for specific events or data changes

Steps to set up a data layer

1) Loading the data layer script
2) Declare the adobeDatalayer array

Once above steps has been configred, we can work on various push menthods(  Push the Data object/ Push Event Object/Push functions) Events (registering & unregistering)

The AEM Core Components are availed with Adobe Client Data Layer(Disabled by default - we need to enable it if we have plans to use it).

To push the data from website to Analytics, we need Adobe Client Data Layer extension & Adobe Analytics, Core Extensions to be configured.

Some of the use cases
1) Retrieving Analytics data & using it for Personalization
2) Trigger an update event on page when the stock market value changes

 Reference URLs:

 Adobe client data layer - https://github.com/adobe/adobe-client-data-layer

Wiki - https://github.com/adobe/adobe-client-data-layer/wiki

All you need to know about Project Firefly

 What is Project Firefly? 

Project Firefly is a run time framework for building 3rd party cloud native applications that extend the functionality of Adobe experience Platform(AEP) and Adobe experience cloud.

It provides everything we need to develop an application, even this is extendable - which grows with the needs.

What it contains?

  • Adobe I/O runtime - which is a server-less foundation for running 3rd party custom code on Adobe infra. It provides scaling in /out etc.
  • CLI & SDK - Enables local development, CI/CD. Streamlined way for developers to interact with core Adobe services and automated process.
  • Spectrum(Adobes design language) UI Framework - A React based UI framework for creating experiences that feels native comfort
  • Custom events - Publish and consume custom events with support for webhooks and journaling
  • Cloud services - a range of services for running managing and optimizing custom digital experiences.(Cloud storage, blob storage,CDN etc)
  • Set of Developer tools - Has UX modeling tools, IDE plugins(code completion) and other tools to aid in testing, debugging  and deploying custom experiences.


Whom it will be helpful?

  • System integration developers - who are typically specialized on integrating and extending Adobe enterprise solutions(AEM, Campaign, Marketo, Magento etc)
  • Enterprise developers - who works with enterprise customers to create business sue case demos etc.


What is the difference between I/O Runtime & project Firefly

Project Firefly is a complete app framework to build custom cloud native Adobe Apps, where in Adobe I/O runtime is a server-less platform for running custom code.

Project Firefly is built on top of Adobe I/O Runtime.

What are all the features of Project Firefly?

  • Storage services - We get all relevant storage services to work with a 3rd party app
  • Debugging - Provides various debugging options
  • Logging - Provides evident logging mechanisms
  • Action templates - Project starter templates
  • UI templates - React spectrum templates to help developers
  • Security - Firefly is highly secured


Some of the typical use cases of Firefly
1) Asset migration in AEM(External DAM to AEM) which are in Gigabytes(bulk uploads) - the task can be offloaded to a headless firefly app that uses content fragment API, so  that normal AEM tasks run without issues.

2) Offloading analytics data to Firefly - Firefly can pull the data from Adobe analytics and save it into DB or make it available for Target

3) Data ingestion monitoring - There are cases where we need to import Huge data into AEP and there could be errors in the process. We can use Firefly to monitor the Adobe i/O events and trigger the other system when something goes wrong to take corrective actions

4) Campaign stand dashboard for monitoring- unlike campaign classic, campaign standard doesnt provide a dashboard to monitor workflows. Using Firefly, we can create SPA that display the status of all workflows in Campaign standard - which helps marketers

Friday, 31 July 2020

Ugrading from older versions of AEM to AEM as a cloud Service - Tools


There are some readily available tools which fasten the process of upgrading older AEM to AEM as a Cloud Service.

Cloud Readiness Analyzer-CRA
  • Cloud Readiness Analyzer report is built using the output of the Adobe Experience Manager (AEM) Pattern Detector)
  • This gives high-level understanding of general upgrade readiness.
  • Helps accelerate the processes of assessing readiness to move from an existing Adobe Experience Manager (AEM) deployment to AEM as a Cloud Service.
  • This tool generates a report that identifies areas of potential refactoring, which is the first step in the transition journey to AEM as a Cloud Service.
  • CRA is supported on AEM instances with version 6.1 and above (CRA is supported on AEM instances with version 6.1 and above)
  • CRA can run on any environment, but it is preferred to have it run on a Stage (Author) environment or clone of the production Author environment
  • CRA can be downloaded as zip and uploaded to AEM instance via Package Manager

Notes: CRA utilizes a system service user account named repository-reader-service(default available in AEM 6.2 and later, For AEM 6.1 we will have to manually create it) to execute the Pattern Detector. Version 6.3 and later we can access CRA via tools -> Operations -> Cloud Readiness Analyzer.
AEM 6.2 prvides a link that generates and downloads the CSV report.
and 6.1 the tool is not functional and only the HTTP interface may be used
    

Analyzer report
-  includes the following categories:
  • Application functionality that must be refactored
  • Repository items that must be moved to a supported location
  • Legacy user interface dialogs and components that must be modernized
  • Deployment and configuration issues
  • AEM 6.x features that have been replaced by new functionality or that are currently not supported on AEM as a Cloud Service
  • The report generation time required is highly dependent on the size and nature of the AEM repository content, the AEM version, and other factors
-----------
Content Transfer Tool CTT

This tool helps to move existing content over from a source AEM instance (on-premise or AMS) to the target AEM Cloud Service instance.
We can use this tool to transfer principals (users or groups) automatically

2 Steps involved
Extraction- extract the source content from AEM and keeping a temporary place called 'migration set'(cloud storage area provided by Adobe - inactive for more than 30 days it gets deleted )

Ingestion - ingesting content from the migration set into the target Cloud Service instance

**Content Transfer Tool creates a local copy of the repository that is later uploaded to the migration set. So ensure enough space is available in local/source AEM

  • Minimum req for CTT is  AEM 6.3 + and JAVA 8. On lower env upgrade your content repository to AEM 6.5 first
  • author AEM will be unavailable during the whole ingestion process.
  • The recommended upper limit for the repository size is 20 Gb
  • CTT can be downloaded as zip and uploaded to AEM instance via Package Manager
  • Access it from  navigate to tools -> Operations -> Content Transfer

FAQ on AEM as a cloud service

1) Which are all the environments available as part of AEM as a cloud service?

Four types of environments available with AEM as a Cloud Service:
  • Production environment - for the business practitioners
  • Stage environment : performance and quality tests before changes to the application are pushed to the production
  • Development environment -  developers to implement AEM applications
  • Demonstration environment : Training , demos, pocs etc - is simplified to a single author node, all others having min 2 author nodes
2) Which are all the types of programs available as part of AEM as a cloud service?

Two types of programs are initially available for AEM as a Cloud Service:
  • AEM Cloud Sites Service
  • AEM Cloud Assets Service

3) What we get as part of AEM as a cloud license?

When we get the license we will have

  • Code repository (Git)  1
  • Baseline image (Sites or Assets)  1
  • Stage and production environment set (1:1) 0 or 1
  • Non-production environments (development or demonstration) 0 to N
  • Pipeline for each environment 0 or 1

**Note:* The author tier will contain all Sites and Assets functionality for all programs, but the Assets programs will not have a publish tier by default

4) What are all the changes in Author, Publish & Replication in AEM as a Cloud Service

Author and publish features:

Both author and publish tiers always accessed via a load balancer. But publish tier, a Continuous Delivery Network (CDN) is always available.

The method of author to publish replication has upgraded now. AEM as a Cloud Service Sling Content Distribution which  allows one to distribute Sling resources between different Sling instances.{The API works at path level and the distribution agents basically enable distribution of specific paths between instances.}This uses a pipeline service run on Adobe I/O, which is external to AEM.

Replication changes - The replication agents used in previous versions of AEM are no longer used or provided, which might impact the following areas of existing AEM projects:
  • Custom workflows that push content to replication agents of preview servers for example.
  • Customization to replication agents to transform content
  • Using Reverse Replication to bring content from publish back to author
5) Do we have all types of runmodes as in previous version of AEM?

The answer is NO.

Run modes that are defined typically include the service (author and publish) and the environment (dev, stage, prod).

Pattern
<service>.<environment_type>

Fr eg: author.dev or publish.prod

The supported runmode configurations in AEM as a cloud service are:
  • config ( The default, applies to all AEM Services )
  • config.author ( Applies to all AEM Author service )
  • config.author.dev ( Applies to AEM Dev Author service )
  • config.author.stage ( Applies to AEM Staging Author service )
  • config.author.prod ( Applies to AEM Production Author service )
  • config.publish ( Applies to AEM Publish service )
  • config.publish.dev ( Applies to AEM Dev Publish service )
  • config.publish.stage ( Applies to AEM Staging Publish service )
  • config.publish.prod ( Applies to AEM Production Publish service )
  • config.dev (*Applies to AEM Dev services)
  • config.stage (*Applies to AEM Staging services)
  • config.prod (*Applies to AEM Production services)

** OSGI configuration that has the most matching runmodes is used.

6) How do we do local development when AEM is in cloud?

For local development:
The following artifacts are made available to the developers:
  • The AEM as a Cloud Service QuickStart: a .jar based, standalone installer of the latest AEM code base, with the same functional and API surface.
  • The AEM as a Cloud Service Dispatcher SDK: an image-based process for testing and validating Dispatcher configurations locally

[* The quickstart is a simple author environment where the majority of the extensions can be developed and tested - does not allow for all AEM Sites and AEM Assets functionalities]

7. What are all some of the important terminologies which we should be aware of?

New important terminologies w.r.t. cloud
  • AEM as a Cloud Service - The cloud-native way of leveraging the AEM applications
  • AEM Image - A deployable artifact that contains the AEM product code together with the customer code.
  • Golden Master - The AEM publish tier.
  • Orchestration Engine - AEM as a Cloud Service uses an orchestration engine to ensure that all author and publish services are scaling as and when needed.
  • Asset microservices - Cloud-based digital asset processing services that cater to various asset processing use cases, such as rendition generation, PDF processions, subasset handling, text extraction etc.
8) What are all the types of users and roles available in AEM as a cloud?

Users & Roles

Cloud Manager currently defines four roles for users which govern the availability of specific features:
  • Business Owner - defining KPIs, approving production deployments
  • Program Manager -  team setup, review status
  • Deployment Manager - execute stage/production deployments
  • Developer - Develops and tests custom application code , do git operations

* There is a role 'Content Author' who does not interact with Cloud Manager. May use Cloud Manager Program Switcher (having navigated from Experience Cloud) to access AEM

9) How do we upgrade from existing AEM version to Cloud Service

> Planning
Access cloud service readiness - determine areas that will require refactoring to be compatible with AEM as a Cloud Service.(Source code Vs deprecated features, New path structure w.r.t AEM as cloud). Then estimate the plan.
Review resource planning - identify resources, create a team, and map out roles and responsibilities
Establish KPIs - Define KPIs to help your team focus on what matters the most.

> Execution
- onboad -familiarize & deploy the code to cloud service
- Integrate - the Git and deploy the code, content transfer, code refactor( Use tools where ever possible For eg: Asset Workflow Migration Tool, Dispatcher Converter, Modernization Tools. )
- Configure - user roles and other things on Admin console of cloud manager

> Post Go-Love
clean-up of temporary files,
review best practices for continuous development
manage logs

10) What are all the options available to troubleshoot anything in AEM as a cloud service?

The following tools are available to troubleshoot AEM as a Cloud Service environments:
  • Developer Console
  • CRX/DE Lite
  • Managing Logs

11)  What does the SDK For Local Development contains in AEM as a Cloud Service?

SDK is comprised of the following artifacts
  • Quickstart Jar - The AEM runtime
  • Java API Jar - all allowed Java APIs that can be used to develop against AEM as as Cloud Service(Previously known as Uber Jar)
  • Javadoc Jar - The javadocs for the above JAR
  • Dispatcher Tools - set of tools used to develop against Dispatcher locally
12) How does the Maintenance tasks became more users friendly now?

With AEM as a Cloud Service, the need for customers to configure the operational properties of maintenance tasks is minimal. Customers can focus their resources on application-level concerns, leaving the infrastructure operations to Adobe.

Below given customer owned the configuration
  • Ad-hoc Task Purge
  • Workflow Purge
  • Project Purge

13) Can you explain mutable vs immutable in AEM as a Cloud Service?

Mutable Vs immutable
  • /apps and /libs are considered immutable areas of AEM as they cannot be changed (create, update, delete) after AEM starts (i.e. at runtime). Any attempt to change an immutable area at runtime will fail.
  • Everything else in the repository, /content , /conf , /var , /home , /etc , /oak:index , /system , /tmp , etc. are all mutable areas, meaning they can be changed at runtime.
  • Oak indexes are mutable at run time
  • *  /oak:index configurations are part of the Code Package and not part of the Content Package