transfer learning and domain adaptation

why we need transfer learning and domain adaptation?

traditional DL:
- isolated for separate tasks
- needs large data
- computationally expensive
- models are static after training (the world changes)
challenges:
- poor performance with limited data - overfitting / under-fitting / poor generalization
- poor domain transfer - model for one domain performs bad in other domain
- evolving data - changes in env results in model’s degradation
the vicious cycle:
- annotated data → model depl → model obsolete → data collection → annotated data

poor performance with limited data
lacking domain transfer
evolving data landscapes

transfer learning

reusing the knowledge from a pre-trained model to solve a new task!
step-1 - pre-train a model on a related task with lot of data ex- AlexNet
step-2 - apply to target task - feat extraction or fine-tuning:
- feat-extraction - frozen model + trainable head
  - use pre-trained model to extract generic feat
  - train n/w on top of the model called head on the target task using these feat
  - ideal when the target data is very limited
- fine-tuning - partially trainable model + fully trainable head
  - train the pre-trained model on the specific target task
  - apply past knowledge from src and relearn the target domain
  - ideal when the target data is large
transferable knowledge - info, patterns, representations
types of transferable knowledge
- low level feat
  - edges, color, word embedding
  - fundamental characteristics
- high level semantics
  - abstract and complex concepts
  - images - recognizing obj or context
  - text - understanding sentiment
- domain specific
  - specific to a particular domain → improves performance on same task
- generic knowledge
  - not specific to a particular task → can be transferred to a wide range of tasks and domains
  - ex - ImageNet pre trained model - semantic seg, depth est, pose est
- high task similarity
  - ex - truck recog and car recog
- low task similarity
  - ex - speech recog and obj det
- high domain sim
  - dataset with similarity
- low domain sim
  - distinct datasets
when to use transfer learning?
- insufficient data
- limited compute
- time constraints
efficient fine tuning techniques
- quantization - quantize model wt to lower precision for more efficiency!
- Low-rank adaptation (LoRA) - freeze the original model wt and instead train so called rank decomposition matrices
- QLoRA - quantization + low rank adaptation

initially A is a gaussian dist and B=0 making delta BA=0
The idea is essentially breaking down the larger d x d weight matrix into smaller matrices d x r and r x d so that the number of params that need update reduces!
initially without LoRA we need to update ⇒ d x d params
with LoRA we need to update ⇒ d x r + r x d << d x d when r is a small number!
Benefits of LoRA:
- maintains a high model quality and does not introduce and additional latency during inference!
- reduces storage requirements, task-switching overhead, and hardware barrier
- agnostic to the training objective and orthogonal to many other methods
Limitations:
- sensitive to learning rate selection and introduces new hyperparameters such as rank and alpha which require careful tuning
- struggles with complex, novel tasks, widening performance gap when compared to full fine-tuning

Yes, since the domain is the same ie. medical images we can use the pre-trained model to improve performance
No, this will very unlikely help as there is low domain similarity

questions like these in exam! - scenario based

types of transfer learning

inductive transfer learning
transductive transfer learning
unsupervised transfer learning
inductive
- labeled target
- domain - same, task - diff
- multi-task learning
  - labelled src + target
  - simultaneously learns many tasks and generalizes well
- self-taught learning
  - unlabeled src, labelled target
  - src and target task may be diff
- ex- pre trained BERT for sentiment analysis, pre trained AlexNet for img segmentation
- benefits
  - reduces need for large labeled dataset for target task
  - speeds up training process
transductive
- labeled src
- domains - diff, task - same
- ex- sentiment analysis in english → french, synthetic → real data
- benefits:
  - allows use of models in new domain without labeled data
  - enhances generalization
unsupervised transfer learning
- unlabelled src
- unlabelled target
- ex- autoencoders , clustering, metric learning
- reduces need for extensive labels, saving time and resources, improves generalization

how to remember?

exam tip:- determine task, domain similar, and which learning to use?

transductive learning - labelled src and unlabelled target, domains diff but sim task ex- sentiment analysis in english → unlabelled french data!
self-taught - type of inductive learning which leverages unlabeled data from the src domain to learn useful feat. It does not require the src and target tasks to be the same and applies the learned feat to the target task, which has labeled data

domain adaptation

problem - src domain and target domain can differ in dist called domain shift and we must try to reduce this gap!
solution - domain adaptation improves performance on target domain from src domain for the same task by reducing domain shift
types of domain adaptation:
- unsupervised (transductive) - labelled src domain and unsupervised target domain
- semi-supervised - some labelled data for target
- supervised - inductive ie. labeled target and unlabeled src
unsupervised - src and target has some overlap
- problems - src is biased towards src dist and does not perform well in target - poor generalization and overfitting! aka sample selection bias
- solutions:
  1. importance sampling
    - works when src sufficiently covers target domain
    - give more imp to examples that represent target domain and low imp to ones representing src domain
    - train a binary classifier c to discriminate between src and target data
    - compute importance weight w(x) = (1-c(src|x))/c(src|x)
    - reweight or resample the src data according to w(x)
    - train the task classifier using the reweighted or resampled data
  2. feat alignment - Domain Adversarial Network
    - incase src does not cover target dist
    - the encoder encodes src data and target data
    - we try to fool a domain classifier c → if the samples are indistinguishable to the discriminator then the dist are aligned
  - 2 ways to update the domain classifier:
    - gradient reversal
      - acts as an identity function in fwd pass
      - in bwd pass multiplies the gradient by a neg scalar
      - feat extractor produces indistinguishable features for the domain classifier
    - optimize for 50-50 guessing
      - optimize the feat extractor to make the domain classifier output probabilities close to 0.5 for both src and target domains
      - this forces the feat extractor to produce feat that the domain classifier cannot distinguish, aligning the source and target feature distributions.
  - advantages of feat alignment:
    - simple to implement and can work well
    - does not require src data dist to cover target data dist
  - disadvantages of feat alignment:
    - requires clear alignment (if the src and target are very different dist, finding a common representation may be difficult or harmful aka negative transfer)
  1. Transferring domain style
    - when it is hard to align feat we translate between domains by learning a mapping function F (that maps src samples to target samples) and a function G (that maps target samples to source samples)
    - steps (same can be done vice versa on target):
      - translate labeled src data → target with func F and train the predictor on src
      - train the predictor on the translated dataset use G to translate target to src
      - deploy pred in the target domain and evaluate pred on translated sample
      - but how to learn G and F?
        
        we use GAN where the generator G tries to generate the src image to a target like image and the discriminator tries to distinguish between the real and the translated image, and eventually the generator gets better with training!
      - challenges:
        
        the mapping could be just arbitrary and inconsistent due to insufficient constriaints
        
        G(x) may not preserve the semantic content of x meaning the translated image G(x) may not correspond to the original image!
      - solution - Cyclic consistency
      - applications:
        
        Robotics Sim2Real - robot sim ←> real robot
        
        human-robot domain adaptation - human images ←> robot images
        
        CyCADA incorporates both cycle consistency and domain adversarial training!
- the underlying task must remain the same while data distribution between src and target might be different
- a technique used in adversarial training particularly in the context of domain adaptation

continual learning

agent must learn over time and adapt in a non-stationary environment, where it encounters a sequence of tasks that must be learned and addressed sequentially without forgetting previously learned tasks!
challenges + objectives:
- limited access to prev data
- minimal increase in the model capacity and computation
- fast adaptation and recovery
- minimize catastrophic forgetting and interference
- maintaining plasticity
- maximize fwd and bwd transfer
- stability and plasticity dilemma
these points are competing against each other as maintaining perfect recall in a fixed-capacity model is impossible
fast adaptation competes with stabilization (stability-plasticity dilemma in the brain)
Catastrophic Forgetting
- use an independent model for every task (needs storage :()
- share parts of network structure across tasks - catastrophic forgetting
Continual Learning
- regularization based - reg terms added to balance old and new tasks
  - weight reg
  - fn reg
- replay based - approx and recover old data dist to mitigate catastrophic forgetting
  - experience replay
  - generative replay
  - feat replay
- optimization based - explicitly design and manipulate optimization
  - gradient proj
  - meta-learning
- architecture based - construct task specific params to mitigate inter-task interference
  - param allocation
  - model decomposition
  - modular network
storage and catastrophic forgetting

Exercise

learn more on this regularization based methods exact replay is also known as experience replay

Note: 1 byte = 8 bits → 7B model in 16 bits → 7 billion x 2 bytes = 14GB since there is a gradient respect to every weight, gradient → 14GB Optimizer - Adam tracks momentum and variance for each param so → 2 x 14 = 28GB total = 56GB + activations ~ 60GB