In case you missed the launch announcement - skills are structured markdown files with (optional) resources that can augment LLM’s with additional capabilities (eg. create a power point, run a financial analysis, etc).

A skill is defined by a SKILL.md file with specific frontmatter. A full functioning example below:

---
name: hello
description: A greetings skill. Invoke when the user gives salutations.
---

Enthusiastically say hi to the users. Use lots of emojis!

This skill can be further extended by bundling additional files with the main skill file.

hello/
├── SKILL.md (main instructions)
├── EMOJIS.md (catalogue of emojis to use)
└── scripts/
    └── animate.py (utility script that can animate emojis)

Skills have completely taken over the majority of my llm based workflows and its hard now to imagine working without them. They represent the most exciting step forward for augmenting LLM capability (in user space) since the introduction of MCPs (but with real use cases).

Priors

Prior to skills, every new conversation with a LLM felt like a scene from 50 first dates.

Yes, we had standards like AGENTS.md and MCPs to help customize LLMs. The former helped steer LLMs using a global README file, the latter helped augment LLM capability through the use of arbitrary tools. But neither approach was scalable as the number of workflows and tools kept increasing.

Stuffing everything in a giant AGENTS.md file eventually leads to context issues and impacts model performance on unrelated tasks. MCPs helped encapsulate steps in a workflow but introduced the additional overhead of requiring code execution. They also run into context issues as the number of MCPs increase because the LLM will load the entire function signature of each MCP (some of which can consume thousands of tokens) into context at the start of every conversation.

Skills

Enter skills. Because it is just a markdown file, it inherits the simplicity of the AGENTS.md file. Because the LLM only reads the title and description instead of the entire skill in context when starting a conversation, it is extremely token efficient. And because skills can bundle in code and additional resources, it means you get the full power of MCPs as well as anything else you care to bundle.

If we think of prompts as procedural code (execute said logic from top to bottom) and MCPs as function calls (encapsulation of related logic), then skills represent object oriented programming.

Skills give prompts the necessary structure to let us compose existing LLM primitives into larger and more ambitious constructs. This means end users no longer need to wait for model providers to train smarter models to unlock additional capability - they can build skills instead.

What’s Next

Skills are great but they do have some limitations. Most notably, they are only available for Claude and have no built-in sharing mechanism. I ended up building a CLI (skillz) to solve the availability issue for my own workflows - it works by injecting skill instructions into codex and other LLM powered coding tools.

Just as object-oriented programming transformed software development by making code reusable and composable, skills represent a similar inflection point for LLMs. We’re moving from a world where new capabilities required model retraining to one where users can extend and specialize their AI tools through composition. The next few years won’t just be about bigger models—they’ll be about richer ecosystems, better sharing mechanisms, and new abstractions we haven’t even imagined yet.

If you’re still copy-pasting prompts or struggling with context limits, consider trying out skills.

In a distributed system, state is akin to original sin - we can't help having it somewhere and it causes all sorts of problems because it's there. This is especially true for distributed state - that is context that needs to be shared across multiple nodes in a system. In this post, we dive into how we manage state at Nimbus by leveraging Vector, a high-performance tool for building observability pipelines, to optimize log aggregations while ensuring availability and scalability.

Context

Nimbus help customers optimize their log volume by aggregating similar logs into an aggregate log. During this process, Nimbus reduces, on average, log volume by 80% and log size by 30%. When aggregating logs, a subset of aggregations are stateful - that it to say it matters what node an aggregation is routed to.

Take the simplified example of logs with a common trace id:

{trace_id:1, msg: start checkout, timestamp: 1}
{trace_id:1, msg: apply discount, timestamp: 2}
{trace_id:1, msg: verify payment, timestamp: 3}
{trace_id:1, msg: complete transaction, timestamp: 4}

The same logs after stateful aggregation (aka all logs being processed by the same node):

{trace_id:1, msg: [start checkout, apply discount, verify payment, complete transaction], timestamp: 1, timestamp_end: 4, size: 4}

Note that during the aggregation, we drop the individual timestamps and keep just the start and end timestamp of the group. If aggregation were not stateful, it can result in the logs being split across multiple aggregations.

An example of the same logs without stateful aggregation:

{trace_id:1, msg: [apply discount, verify payment ], timestamp: 1, timestamp_end: 3, size: 2}
{trace_id:1, msg: [start checkout, complete transaction], timestamp: 2, timestamp_end: 4, size: 2}

Not only is the latter result less efficient in terms of size and volume reduction, it also causes operational issues. Because logs are now interleaved between groups, it is hard to see the proper sequence of events and make it difficult to do log based monitors and dashboards.

One way of getting around this problem is to run a single node to do all aggregations. This can get you quite far but come with scalability, availability and durability issues due to having a single point of failure. Another solution is to accept sub-optimal log aggregations and letting any node manage any aggregation. But if neither of these options are acceptable, then the system itself needs to coordinate traffic so that logs that belong to a common aggregation are processed by the same node. This system is what we'll be describing in this post.

Nimbus Architecture

A high level overview of the Nimbus ingestion pipeline consists of the following components

                                  ---------  Vector Fleet ----------
Load Balancer -> Router Fleet -> | Buffer Fleet -> Aggregator Fleet |

Traffic is fronted by an application load balancer. This is forwarded to our routing fleet which is responsible of directing customer traffic to the correct vector fleet - each customer has a separate fleet of buffers and aggregators.

Buffers are responsible for ingesting data and writing (buffering) it to disk before forwarding them to the aggregator. Aggregators are responsible for aggregations. All services are deployed across multiple availability zones and are orchestrated using kubernetes via EKS

The buffer fleet is a fleet of vector instances that have both the aggregation rules as well as the aggregator endpoints of a given customer.

Aggregations

Aggregations are defined in YAML and are built on top of vector's reduce transform.

An example of an aggregation rule:

id: cd0763c8-c4b5-4a61-aa14-51bcdee62db2
process_when: <list of predicates that if true, qualify a log for aggregation>
group_by: <optional list of keys that aggregation will be grouped by>
merge_strategies: <optional key specific strategies for aggregating keys>
...

To implement stateful aggregation, we need to ensure logs belonging to an aggregation groups always get aggregated by the same aggregator. To identify logs that belong to the same aggregation group, we create a hash of their id and group_by values to uniquely identify an aggregation. We then rely on consistent hashing to route traffic to the correct aggregator. The set of aggregators is initially hardcoded doing service bootstrapping but can change due scaling events. This means we also need a mechanism of service discovery so that buffers can route traffic correctly after scaling.

Neither service discovery or consistent hashing are built in features for vector, so to make this possible, we (ab)use the Vector Remap Language (VRL) to carry out the logic.

Consistent Hashing

Consistent hashing is a specialized hashing technique that minimizes work when resizing hash slots. It works by mapping hash slot as points in a circle. When a new value is added, its hash is mapped to the circle and it is grouped to the nearest preceding slot.

The following is what consistent hashing might look like for an aggregation with a single group_by key: trace_id.

transforms:
  route/l/checkout_pre:
    type: remap
    source: |
      rule = "cd0763c8-c4b5-4a61-aa14-51bcdee62db2"
      point = abs(mod(seahash(join!([rule, to_string!(.trace_id)])), 360))
      if (point >= 0 && point < 120) {
        .__nimroute = "agg0"
      } else if (point >= 120 && point < 240) {
        .__nimroute = "agg1"
      } else {
        .__nimroute = "agg2"
      }
  out/l/router:
    type: route
    route:
      agg1: .__nimroute == "agg0"
      agg2: .__nimroute == "agg1"
      agg3: .__nimroute == "agg2"
    inputs: 
      - route/l/checkout_pre

First, the join function concatenates the rule name and the trace id value into a single string which is then hashed into a number using seahash. We then modulo the result by 360 and take the absolute value to get to a point in the circle. The series of if/else statements afterwards "map" that point to the appropriate "slot", in our case, this is assigning a custom property to the log called __nimroute. Finally, the __nimroute parameter is then used by the route component to create different routes which will be used to route traffic to a specific aggregator.

Service Discovery

The Nimbus configuration service is responsible for generating vector configuration with the consistent hashing logic as well as aggregator endpoints. Because aggregators run as k8s statefulsets, they have stable network ids. This means that endpoint information can be constructed based on the aggregator naming scheme $CUSTOMER_ID-agg-$ORDINAL.vector.svc.cluster.local and the number of replicas in the stateful set.

Javascript code for getting service endpoints:

const services = [];
for (let i=0; i <REPLICAS; i++) {
  services.push(`${CUSTOMER_ID}-agg-${i}.vector.svc.cluster.local`)
}

The following are sinks generated for our current example with three replicas.

sinks:
  sink/a/agg0:
    type: vector
    address: $CUSTOMER_ID-agg-0.vector.svc.cluster.local
    inputs:
      - out/l/router.agg0
  sink/a/agg2:
    type: vector
    address: $CUSTOMER_ID-agg-1.vector.svc.cluster.local
    inputs:
      - out/l/router.agg1
  sink/a/agg2:
    type: vector
    address: $CUSTOMER_ID-agg-2.vector.svc.cluster.local
    inputs:
      - out/l/router.agg2

Etc

Note that in this system, aggregators still represent a single point of failure for the set of logs that they are responsible for. In order to mitigate this, we rely on aggregators buffering data to disk (synced acrossed multiple availability zones using EFS) and having buffers retry sending data until aggregators come back online. This works for temporary restarts but can be problematic for longer outages due to fixed buffer sizes on the buffers. We have a secondary mechanism of also buffering data into s3 and a replay mechanism to re-send the data for the latter case. More details on this in a future post.

Final Thoughts

In distributed systems, managing state is often a necessary evil to deliver useful results. By combining dynamic config generation and the versatility of VRL, it is possible to implement well-known techniques such as consistent hashing and service discovery within Vector without the need for external services. Even if state can't be entirely avoided, we can design systems that minimize complexity when handling it.

Observability data, when left on its own, behaves like entropy - it increases without bounds (until you run out of usable energy money).

This is especially apparent this last decade with the rise of kubernetes and micro services which emit orders of magnitude more telemetry than the humble monoliths that came before it. With the increase in data volume comes the need for new tools to manage it. Enter the observability pipeline.

Observability pipelines helps organizations collect, transform, and send telemetry from any source to any destination. They are becoming increasingly popular among the enterprise because they allow organizations to control their data and limit volume before sending it upstream to vendors.

Most pipeline solutions rely on YAML for configuration. Some have a GUI that helps with generating YAML. But regardless how you do it, managing complex pipeline topologies using YAML quickly becomes error prone and hard to scale.

This post goes over a step by step example of building a scalable pipeline configuration. We'll start with a basic example of sending and receiving data and build upon it layer by layer to get to a pipeline that can handle any number of components and branching topologies.

Vector

For the purposes of this post, we will be writing pipeline configuration for Vector, a high performance pipeline written in rust. We'll also limit the examples to working with log data. The general principals in this post will be applicable to other pipeline implementations and signals.

1 - Send and Receive

This is a basic pipeline configured to send and receive data. In this case, its collecting logs from the datadog agent and sending it to the datadog vendor destination.

sources:
  source/a/dd_agent:
    type: datadog_agent
    address: 0.0.0.0:8020
    multiple_outputs: true
sinks:
  sink/l/dd:
    site: ${DD_SITE}
    type: datadog_logs
    inputs:
      - source/a/dd_agent.logs
    compression: gzip
    default_api_key: ${DD_API_KEY}

A few things to note about this configuration:

• vector lets you dynamically inject environmental variables to the configuration. we use this to add add secrets and configurable values such as the datadog site and api key

• we're using a deliberate naming scheme for pipeline components. The convention is $stage/$signal/$detailed_name.

  • $stage describes the pipeline stage. a source receives data. a sink sends data.

  • $signal denotes the type of data. a applies to all data. l. is logs. m is metrics. t is traces

  • $detailed_name gives further detail about the $stage or the function of the particular component

2 - Basic Pipeline

The previous example was somewhat contrived since you could just send logs via an agent without using a pipeline. This example adds a transformation step to shape the data before sending it upstream.

sources:
  source/a/dd_agent:
    type: datadog_agent
    address: 0.0.0.0:8020
    multiple_outputs: true
transforms:
  in/l/remap:
    type: remap
    inputs:
      - source/a/dd_agent.logs
    source: |
      .message = parse_json(.message) ?? .message
sinks:
  sink/l/dd:
    site: ${DD_SITE}
    type: datadog_logs
    inputs:
      - in/l/remap
    compression: gzip
    default_api_key: ${DD_API_KEY}

This particular transformation will try to parse the message field as json. If it fails, it will leave the message field as is. Note that this transform applies to all incoming logs and has the in/ $stage prefix.

3 - Routing and Aggregation

In real world deployments, you'll tend to have more than one transformation. You'll also not want to apply all transformations to all data all the time. This example adds a second transformation that is conditionally applied via a router component.

sources:
  source/a/dd_agent:
    type: datadog_agent
    address: 0.0.0.0:8020
    multiple_outputs: true
transforms:
  in/l/remap:
    type: remap
    inputs:
      - source/a/dd_agent.logs
    source: |
      .message = parse_json(.message) ?? .message
  in/l/route:
    type: route
    route:
      itemrefresh: (.service == "itemrefresh")
    inputs:
      - in/l/remap
  route/l/itemrefresh: 
    type: reduce
    inputs:
      - in/l/route.itemrefresh
    group_by:
      - host
    max_events: 300
sinks:
  sink/l/dd:
    site: ${DD_SITE}
    type: datadog_logs
    inputs:
      - in/l/route._unmatched
      - route/l/itemrefresh
    compression: gzip
    default_api_key: ${DD_API_KEY}

In this configuration, the router is forwarding logs where service = itemrefresh to the reduce component for aggregation. Router components have a router/ $stage prefix. Non-matching logs are sent to the _unmatched path. All log paths are then fed into the datadog sink for sending to datadog.

Vector provides different merge_strategies for aggregating individual message fields. In this case, we rely on the default behavior, which keeps the first string field as well as the first and last timestamp field in the aggregated output.

4 - Multiple Routes

This expands on the previous example and adds a second route that forwards logs to a different reduce component.

sources:
  source/a/dd_agent:
    type: datadog_agent
    address: 0.0.0.0:8020
    multiple_outputs: true
transforms:
  in/l/remap:
    type: remap
    inputs:
      - source/a/dd_agent.logs
    source: |
      .message = parse_json(.message) ?? .message
  in/l/route:
    type: route
    route:
      itemrefresh: (.service == "itemrefresh")
      checkout: (.service == "checkout")
    inputs:
      - in/l/remap
  route/l/itemrefresh: 
    type: reduce
    inputs:
      - in/l/route.itemrefresh
    group_by:
      - host
    max_events: 300
  route/l/checkout: 
    type: reduce
    inputs:
      - in/l/route.checkout
    group_by:
      - customerId
      - transactionId
    max_events: 300
    merge_strategies:
      customerId: discard
      transactionId: discard
sinks:
  sink/l/dd:
    site: ${DD_SITE}
    type: datadog_logs
    inputs:
      - in/l/route._unmatched
      - route/l/*
    compression: gzip
    default_api_key: ${DD_API_KEY}

Vector accepts adding wildcards to route names which makes it possible for us to combine data from all routers into a single path route/l/*. Because each component has the signal type as part of its identifier, its easy to select components on a per signal basis.

5 - Complex Routes

In some cases, you might need to apply multiple transformations to some data. In this example, we pull up some fields nested under message to the top level. This is actually required as we use these fields in the group_by clause of the reduce component and vector does not support grouping by nested fields.

sources:
  source/a/dd_agent:
    type: datadog_agent
    address: 0.0.0.0:8020
    multiple_outputs: true
transforms:
  in/l/remap:
    type: remap
    inputs:
      - source/a/dd_agent.logs
    source: |
      .message = parse_json(.message) ?? .message
  in/l/route:
    type: route
    route:
      itemrefresh: (.service == "itemrefresh")
      checkout: (.service == "checkout")
    inputs:
      - in/l/remap
  route/l/itemrefresh_exec: 
    type: reduce
    inputs:
      - in/l/route.itemrefresh
    group_by:
      - host
    max_events: 300
  route/l/checkout_pre: 
    type: remap
    inputs:
      - in/l/route.checkout
    source: |-
      .transactionId = .message.transactionId
      .customerId = .message.customerId
      del(.message.transactionId)
      del(.message.customerId)
  route/l/checkout/exec: 
    type: reduce
    inputs:
      - route/l/checkout_pre
    group_by:
      - customerId
      - transactionId
    max_events: 300
    merge_strategies:
      customerId: discard
      transactionId: discard
sinks:
  sink/l/dd:
    site: ${DD_SITE}
    type: datadog_logs
    inputs:
      - in/l/route._unmatched
      - route/l/*exec
    compression: gzip
    default_api_key: ${DD_API_KEY}

Note that we change the naming convention for components after the router. Pre-processing transformations have the _pre suffix appended and the main transformation has the _exec suffix. This lets us continue to aggregate all the routes using a single path route/l/*exec in the sink.

Review

At this point, we've built a pipeline configuration that is both flexible and easy to reason about.

1. Global transformations can be applied by adding them at the beginning of the pipeline

2. Scoped transformations can be applied by adding them after a router component

3. Additional transformations can be added using suffixes before and after other transformations

4. All inputs and outputs are well defined with well structured names that make it easy to wire up and diagnose

5. Paths from any stage can be aggregated by using $stage/* and can be further filtered by signal $stage/[l|m|t]/*

The well defined naming scheme also makes it easy to do diagnostics. Vector emits the component identifier as a label on all metrics which means that it is now easy to graph related metrics in the same dashboard. The following are examples of some helpful queries written in PromQL syntax:

• get metric from all sources: $metric_name{component_id=~"source/*"}

• get metric from all components dealing with logs: $metric_name{component_id=~".+/l/.+"}

• get metric from all routing pre-processing components: $metric_name{component_id=~".+/.+/.+_pre"}

Lastly, Vector makes it easy to capture the inputs and outputs of different components and inspect data flows via vector tap $component_id. This command also takes wildcards which means that you can easily tap components by $stage and $signal.

An example of tapping all transformations that are scoped to a route:

vector tap --outputs-of "route/l*"

Additional Topics

For the sake of brevity, we left out everything from this post that doesn't directly relate to managing configuration.

Other important topics:

• buffers, batches, and timeout configuration for optimizing data durability

• power and perils of using custom transformations via scripting languages like VRL and OTTL

• high available pipeline deployment topologies

• day two pipeline operations - zero downtime updates and migrations

Look out for future posts that will dive further into these topics.

Final Thoughts

Pipelines are a recent but powerful addition to the modern observability stack. They help organizations limit vendor lock in and take full control over their observability data.

If you want to have the benefits of a pipeline without the toil of operations, consider Nimbus, a fully managed observability pipeline that analyzes your traffic and automatically optimizes your data. On average, Nimbus helps customers reduce log volume by 60% without dropping data.

At the end of the day, observability data should help you fight fires, not be the cause of internal fire fighting - if you find yourself on the wrong end of that equation, consider updating the plumbing and adopting a pipeline.

On a rainy March day in 1538, Thomas Howard, the Duke of Norfolk, found himself confined within the cold stone walls of his grand estate. His fingers reluctantly penned a letter to sell his cherished lands to settle longstanding debts.

"A man can not have his cake and eat his cake", he wrote, the words etched with the pain of having to confront a reality that he did not want to accept.

This was the earliest written record of the proverb "you can't have your cake and eat it too". It's used in situations when we are forced to make tradeoffs that we'd rather not make.

When it comes to logging, that tradeoff is one of volume vs cost. You want the volume necessary to observe all your systems and you want to not spend most of your infra budget on logs. But you often can't have both.

Not Eating Cake

The common choice companies make when logging costs are high is to log less. It lowers the cost and also reduces the volume.

This could be deciding that you only need ERROR logs in production. Or deciding that certain services can do without logs at all. Or only keeping logs on the host and hoping it doesn't crash.

While this can reduce costs, it's often a temporary bandage, one with very real downsides.

Not Having Cake

The reason companies log in the first place is because logs are insurance. In the happy case, no one thinks about it (besides the CFO griping about the bill). In the other case, the one where things are on fire, logs are essential for engineers to properly diagnose and recover from an incident.

Incidents result in downtime and downtime impacts the business's bottom line. When Amazon went down for 40min in August 2013, a Forbes article estimated that Amazon lost $66,240 per minute during the outage

Dropping logs does decrease your costs. But you need to balance these savings with additional costs incurred by every extra second of downtime.

Have it Both Ways

What if you didn't need to choose? What if you could store all your logs while lowering your costs? Enter Lossless Log Aggregation (LLA).

LLA is the process of aggregating similar logs into a larger aggregate log. Common metadata and values are deduplicated and merged during the aggregation. When done effectively, this can result in a 100X reduction in volume and a 40% reduction in size. Without dropping data.

Illustration of LLA

Types of Log Groups

Three types of log groups are good candidates for LLA:

1. Logs with common message patterns

2. Logs with common identifiers

3. Multi-line Logs

To see before and after examples of each type, see the examples section of the Nimbus Documentation.

Logs with common message patterns

These are high-volume log events that repeat most of their content. For most applications most of the time, this will be the primary driver of log volume. Examples include health checks and heartbeat notifications.

Logs with common identifiers

These are logs that describe a sequence of related events. These sequences usually have some sort of common identifier like a transactionId or a jobId. Examples include a background job and business-specific user flows.

Multi-line Logs

These are logs where the message body can be spread across multiple new lines. Unless you add special logic on the agent side, the default behavior is to emit each newline delimited message as a separate log event.

A Motivating Example

Below are logs from a load balancer performing health checks on a suite of targets.

{"timestamp":"Tue Jan 10 09:15:16 2023","host":"lb1.nimbus.dev:5678","target":"web1","path":"/health","latency":"23ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:15:18 2023","host":"lb2.nimbus.dev:5678","target":"web2","path":"/health","latency":"57ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:15:25 2023","host":"lb1.nimbus.dev:5678","target":"web4","path":"/health","latency":"14ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:15:28 2023","host":"lb2.nimbus.dev:5678","target":"web5","path":"/health","latency":"38ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:16:01 2023","host":"lb3.nimbus.dev:5678","target":"web3","path":"/health","latency":"16ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:17:16 2023","host":"lb1.nimbus.dev:5678","target":"web1","path":"/health","latency":"19ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:17:18 2023","host":"lb2.nimbus.dev:5678","target":"web2","path":"/health","latency":"41ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:17:22 2023","host":"lb3.nimbus.dev:5678","target":"web3","path":"/health","latency":"32ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:17:25 2023","host":"lb1.nimbus.dev:5678","target":"web4","path":"/health","latency":"27ms","status":"passed"}
{"timestamp":"Tue Jan 10 09:17:28 2023","host":"lb2.nimbus.dev:5678","target":"web5","path":"/health","latency":"62ms","status":"passed"}
// ... more logs

These are the same logs after applying LLA based on host and status:

{"host":"lb1.nimbus.dev:5678","status":"passed","path":"/health","data":[{"target":"web1","time":"23ms"},{"target":"web4","time":"14ms"},{"target":"web1","time":"19ms"},{"target":"web4","time":"27ms"}, ...], "size":700, "timestamp":"Tue Jan 10 09:15:16 2023","timestamp_end":"Tue Jan 10 09:17:28 2023"}
{"host":"lb1.nimbus.dev:5678","status":"passed","path":"/health","data":[{"target":"web1","time":"23ms"},{"target":"web4","time":"14ms"},{"target":"web1","time":"19ms"},{"target":"web4","time":"27ms"},...], "size": 700, "timestamp":"Tue Jan 10 09:15:16 2023","timestamp_end":"Tue Jan 10 09:17:28 2023"}
{"host":"lb2.nimbus.dev:5678","status":"passed","path":"/health","data":[{"target":"web2","time":"57ms"},{"target":"web5","time":"38ms"},{"target":"web2","time":"41ms"},{"target":"web5","time":"62ms"},...], "size": 700, "timestamp":"Tue Jan 10 09:15:18 2023","timestamp_end":"Tue Jan 10 09:17:28 2023"}
{"host":"lb3.nimbus.dev:5678","status":"passed","path":"/health","data":[{"target":"web3","time":"101ms"},{"target":"web3","time":"16ms"},{"target":"web3","time":"32ms"},...],"size": 700,"timestamp":"Tue Jan 10 09:15:22 2023","timestamp_end":"Tue Jan 10 09:17:22 2023"}
// ... more logs

Some things to note:

• a new attribute, size, has been added to indicate the number of logs aggregated

• a new field, timestamp_end, has been introduced to mark the time of the last log in the aggregation

• a new field, data, holds an array of the unique info of each log

For the above example, we've managed a 77% reduction in log size and a 99% reduction in log volume.

How to do this at Home

LLA can be implemented in an observability pipeline - these are data pipelines that are optimized for receiving, transforming, and sending observability data. Popular open-source pipeline solutions include the OTEL Collector and Vector.

To configure a pipeline for LLA, you'll need to identify log groups, create forwarding rules, normalize log data before and after aggregation, as well as perform the aggregation itself over the correct fields.

You also need to pay attention to vendor-specific details - for example, Datadog has limits for maximum content size, array size, and the size of a single log. This means you need to be mindful of not exceeding any of these quotas when aggregating datadog logs or risk losing data.

Finally, as the pipeline itself is now part of your observability stack, you will need to operationalize it and make sure it can scale to handle traffic from all your services.

If you like the idea of LLA but not necessarily the toil, consider trying Nimbus. Nimbus is the first observability pipeline that automatically analyzes log traffic and can identify high-volume log groups as well as come up with LLAs to reduce their volume. On average, organizations save 60% off their logging costs within the first month of use.

Conclusion

It's 2024, and many companies find themselves at odds with their observability vendor - they want to ship all the logs necessary to provide a reliable service but cannot justify the expense to do so.

Like Duke Thomas, these organizations find themselves having to make the painful tradeoff of needing to part with something they need versus paying for something they cannot in good conscience afford.

But unlike in the Duke's time, there's a third choice - lossless log aggregations. Sometimes you don't have to choose, sometimes you can have your cake and eat it too!

There are certain things in life that I hate doing. Taxes. Getting my teeth pulled. Calculating observability costs.

This post will be focused on the last item, calculating observability cost. Specifically, we'll focus on the usage based pricing schemes of commercial observability vendors. Since observability encompasses such a wide array of services, we'll limit the scope of this post to logging for the sake of brevity. That said, the models we’ll discuss for logging are, in broad strokes, also applicable for metrics and traces.

Pricing Models

When it comes to logs, there are two dominant pricing models employed by the industry:

• Volume based pricing: you are charged in proportion to the total amount of data ingested, usually measured in gigabytes/month.

• Resource based pricing: you are charged in proportion to the amount of resources it takes to process your data, usually measured in compute consumption

These two pricing models can be further split in terms of fixed usage vs variable usage.

• Fixed cost usage: you are charged a higher per unit fee upfront but have no consumption based fee afterwards

• Variable cost usage: you are charged a lower per unit fee upfront but pay a variable amount based on consumption

All together, you can plot vendors along the following two by two grid

Volume Based Pricing

In volume based pricing, you are charged per gigabyte of log (or some gigabyte equivalent). Logs are metered at the following stages:

1. Ingest: This covers the ingestion of the logs as they are sent from your service to the vendor

2. Storage: This covers the storing and indexing of logs as they are processed by the vendor

3. Query: This covers executing search queries by users over the stored logs

Ingest

The ingest stage is generally the cheapest stage of the log processing pipeline. This is because vendors want you to send them more data as this leads to lock in. Ingest to Amazon S3 is free. Ingest to datadog is $0.10/gb.

Many vendors combine ingest and storage. For example, ingest and storage in New Relic starts at $0.30/gb.

Note that there are violators of the rule of cheap ingress. Amazon Cloudwatch Logs charges $0.50/gb for ingest and $0.03/gb/month for storage.

Storage

The storage stage is generally where vendors make the bulk of their margin. Storage is typically measured in gb/month or per million events.

When storing log data, there is a difference between what I refer to as raw storage and indexed storage.

Raw storage is cheaper than indexed storage by an order of magnitude but usually involves variable usage costs at query time. Indexed storage is more expensive but has typically has no variable cost at query time.

Many vendors now also offer different storage tiers where you can pick what portion of your data is indexed.

Below is an updated grid with vendors divided by storage tier. Vendors at the intersection of the dotted line offer different storage tiers or have per seat pricing.

Query

The query stage is generally the hardest to calculate. Vendors like datadog that offer indexed storage bake the cost into their storage fee which means there is no additional query fee.

Vendors that offer raw storage charge a variable cost for searching over the data. This is usually charged by per gb scanned. This isn't an arbitrary charge - the reason that "raw" storage is cheap is because it is not indexed (which makes it cheap to store) and that extra compute needs to be expended to run queries over the data.

Resource Based Pricing

In resource based pricing, you are charged for the underlying processing you consume when working with logs. This is usually measured in terms of raw instances or some abstract compute unit.

In fixed cost usage, you pay for the underlying instances that are responsible for ingest, storage and query. This means you need to determine and allocate enough compute to satisfy your performance, availability, and durability concerns. This involves more work upfront but is usually cheaper, if properly provisioned, than a volume based pricing model. This is because the vendor, for volume based pricing, needs to assume the worst case (100% utilization with maximum availability and durability) while maintaining a margin - few workloads actually consume that in practice.

In variable cost usage, you pay for the underlying compute. For example, Amazon's Serverless Opensearch Service charges customers based on OCUs (OpenSearch Compute Units).

One OCU comprises 6 GB of RAM, corresponding vCPU, GP3 storage (used to provide fast access to the most frequently accessed data), and data transfer to Amazon Simple Storage Service (S3)

Amazon OpenSearch Pricing Docs

AWS will autoscale your compute depending on your usage. You tradeoff needing to manually allocate resources but end up paying a variable premium on the compute.

Price Modifiers

In addition to volume vs resource and fixed usage vs variable usage, there are specific options that you can buy into which multiple your costs across all pricing models.

First, there's the annual plan vs monthly plan. Annual plans lock you in for a certain usage for a year but can be 30% cheaper than the monthly plan. While a 30% discount seems great, effective use of an annual plan involves being able to effectively forecast your log usage across all the factors we've so far discussed. I have yet to see a company at any scale be able to do this effectively.

There's also enterprise discounts at higher volumes. These discounts can be significant but like the annual plan, require hefty upfront commitments.

Second, there's retention - the number of days a vendor will keep your logs indexed. This can be as short as a day to as long as indefinite. The higher the retention, the more it will cost to store the logs.

Finally, there's "value add" options that vendors provide like "sensitive data scanning" and extra compliance guarantees. These options can double the price per gigabyte during ingestion and storage.

Pricing Comparison

At this point, you might have an inkling of why observability pricing is hard to calculate, never mind compare. No two vendors have the same pricing model and they each charge for a different permutation of usage.

Nevertheless, we're going to do a best effort attempt by making a number of assumptions to normalize costs across vendors.

Volume Based Fixed Cost Comparison

The following is a comparison of just the volume based pricing of different vendors in terms of cost per gigabyte per month. It does not cover vendors that use resource based pricing and does not take into account the variable based portion of the cost.

We make the following assumptions when calculating costs:

• for vendors that have a fixed price component ($X for first Y GB and then $Z/gb/month afterwards) - we are ignoring this as it an insignificant part of the cost at scale

• for vendors that have event based pricing ($X per million events), we're assuming that the average event is 512 bytes and normalizing to gb/month

• for vendors that have indexed storage with variable retention prices, we normalize to 14day retention

• for vendors that have multiple tiers of storage (eg. hot and cold), we compare prices for "hot" storage

Some observations:

• indexed storage vendors have the highest price per byte and raw storage vendors have the lowest.

• there is a 33X difference between the most expensive vendor (Datadog at $5.08/gb/mo) and the least (Axiom at $0.15/gb/mo)

• the new generation of vendors that have started within the last 10 years (eg. axiom, signoz, hyperdx, etc. ) have drastically lower volume based prices ($0.50/gb/mo or lower)

Volume Based Variable Cost Comparison

Only looking at fixed cost can paint a distorted picture of the vendor landscape - variable costs can easily exceed the fixed cost depending on the workload.

We make the following assumptions when calculating costs:

• variable usage per month is total team size * volume of data ingested * 10 (this makes the assumption that each team member, on average, queries 10X the monthly ingest volume each month)

• S3 query cost is modeled as the cost of scanning data using Athena ($0.005/gb scanned)

The following is a chart of vendors that have a variable cost during query time. We model a team size of 10 with a monthly data volume of 1TB.

Some observations:

• Cloudwatch Logs cost doubles as the variable cost just about equals the volume cost

• Grafana Cloud has a fair use policy that makes it free to query 100X the monthly volume - this is why you don't see a variable cost component for Grafana at 10 people. Note that if exceeded, Grafana becomes extremely expensive at query time ($0.40/gb scanned)

• New Relic charges per seat - at $418.80/full platform user, the total price increases by over 13X

• S3 query cost becomes the bulk of any logging service built on top of it

Volume Based Combined Cost Comparison

The following two charts attempt to show the total cost of ownership when accounting for both the fixed and variable portions of volume based pricing when processing 1TB of logs per month for a team of 10.

Some observations:

• New Relic has the biggest jump in pricing due to their per seat model

• Cloudwatch logs and S3 become drastically more expensive when taking into account querying

Note that there are many "raw storage" vendors that do not (currently) have a variable based cost associated with querying (eg. Highlight, HypderDX, Signoz, BetterStack). These vendors are all built on top of either S3 (or a cloud vendor equivalent) or Clickhouse (which leverages S3).

This is most likely due to a combination of the following:

• our estimate of queries scaling at 10X monthly ingest volume per user is far too conservative

• the naive assumption of using S3 + Athena to model S3 query is far too conservative

• the aforementioned vendors are able to leverage caching, pre-computation, and other optimizations to drastically reduce the cost of serving queries

• the aforementioned vendors are all "new" to the scene (started within the last 5 years) and don't have customers of the scale and workload that break their current pricing model

Conclusion

Pricing is tricky. Usage based pricing means you pay little when you're small but that your costs can grow exponentially if you scale. This makes it hard to compare or forecast.

If you're just starting out, it probably doesn't matter because costs are minimal and you will get more leverage by focusing on your business. If you're a growing or mature organization and you realize that observability is over 20% of your infra budget or past the seven figure range, it might be worthwhile to look into optimizations (eg. see Datadog's $65M/year customer)

The kicker is that by the time you want to optimize is also when you're most locked in. Existing instrumentation, training, run books, dashboards, alarms - all this and more needs to be migrated at scale without breaking existing services or slowing down product velocity. To switch might mean dedicating a two pizza team for an entire year to deliver an outcome that in all likelihood will be worse than what you have in place today. This is why its worth understanding not just what you're costs are today but how they will scale when you grow.

If you're already in too deep or you just want someone to make the existing costs go away - check out Nimbus (disclaimer: I'm the founder). The Nimbus platform helps companies reduce logging costs by 60% or more with no migrations or code changes. Our optimization engine automatically analyzes incoming logs (metrics and traces available in limited preview) to identify common patterns and create transformations that reduce log volumes without dropping data. If controlling observability costs is a concern, also feel free to reach out directly to kevin(at)nimbus.dev

Abstract of Loki warmings himself with logs, digital art

OpenTelemetry (OTEL) is the industry standard for managing telemetry data. The working group's main focus for the past three years is supporting the three pillars of observability: logs, metrics, and traces.

Today, there is a growing number of organizations that use OTEL for traces and metrics. It is not so for logging. In large part, this is because the logging API was not marked stable until April this year. Most people I talk to are still under the impression that OTEL does not support logs.

Today's post will attempt to correct the misconception. We will be using the NodeJS OTEL SDK to add logging to a simple web app. Logs will be collected using the otel agent (otel-collector-contrib) and exported to Loki, an open-source logging backend. Traces will be generated via auto-instrumentation and auto-correlated with logs (and vice versa). Semantic conventions will be mentioned.

For those who want to jump directly to the code, you can download the result on Github.

Setup the App

First, initialize your nodejs project and install express (a minimalist web framework for nodejs).

mkdir otel-node-demo
cd otel-node-demo
npm init -y
npm i express

Next, write the application code. This application exposes a single endpoint, /rolldice , which will return a random number between 1 and 6.

• app.js

const express = require('express');
const PORT = parseInt(process.env.PORT || '8080');
const app = express();

app.get('/rolldice', (req, res) => {
  const randNum = getRandomNumber(1, 6)
  res.send((randNum).toString() + "\n");
});

app.listen(PORT, () => {
  console.log(`Listening for requests on http://localhost:${PORT}`);
});

Start the application. This will run an express server listening on port 8080

$ node app.js
Listening for requests on http://localhost:8080

In a separate terminal, you can roll the dice by sending a few requests to the /rolldice endpoint

$ curl localhost:8080/rolldice
5
$ curl localhost:8080/rolldice
3

Add Logs

To add logs, start by installing the necessary OTEL dependencies

npm i @opentelemetry/sdk-node \
  @opentelemetry/api \
  @opentelemetry/sdk-logs 

NOTE: The status of the Javascript Logging SDK is currently in development

Next, create an initialization file for the OTEL NodeSDK

• instrumentation.js

const { NodeSDK } = require('@opentelemetry/sdk-node');
const {
  SimpleLogRecordProcessor,
  ConsoleLogRecordExporter,
} = require('@opentelemetry/sdk-logs');

const sdk = new NodeSDK({
  logRecordProcessor: new SimpleLogRecordProcessor(
    // writes logs to stdout
    new ConsoleLogRecordExporter()
),
});
sdk.start();

Our current configuration will write logs to STDOUT. To write logs from our application, we need to acquire a logger instance and emit a log.

• app.js

diff --git a/app.js b/app.js
index 8ee1efe..b916e9a 100644
--- a/app.js
+++ b/app.js
@@ -1,6 +1,8 @@
+const { logs, SeverityNumber } = require('@opentelemetry/api-logs');
+const logger = logs.getLogger('app', '0.1.0');
 
 app.get('/rolldice', (req, res) => {
   const randNum = getRandomNumber(1, 6)
+  const attributes = {
+    'roll': randNum
+  }
+  logger.emit({
+    severityNumber: SeverityNumber.INFO,
+    body: "exiting /rolldice",
+    attributes
+  });
   res.send(randNum.toString() + "\n");
 });

We can now restart our application with the instrumentation code preloaded.

# cancel the running server 
$ CTRL-C

# restart the server while preloading our custom instrumentation
$ node -r ./instrumentation.js app.js

NOTE: The reason we wrote the SDK initialization code in a separate file is to set up instrumentation before loading the main script. This is not required for logging but it is for auto-instrumenting tracing (something we'll cover later in this post).

Sending a request should now yield JSON from the server

$ curl localhost:8080/rolldice

Output from the server shell

{
  timestamp: 1694367487002000,
  traceId: undefined,
  spanId: undefined,
  traceFlags: undefined,
  severityText: undefined,
  severityNumber: 9,
  body: 'exiting /rolldice',
  attributes: { roll: 1 }
}

NOTE: There are lots of fields like traceId and spanId that are currently undefined - these will be automatically filled when we add tracing.

Initialize the Collector

To send telemetry data, we need an endpoint that can ingest the OpenTelemetry Protocol (OLTP). For this guide, we will be using otel-collector-contrib

Download the collector

You can find the latest releases of the collector for all architectures here. In my case, since I'm running on an M1 Macbook, I will be using the otelcol-contrib_0.84.0_darwin_arm64.tar.gz

After downloading, unpack and verify that the collector is working.

$ cd ~/Downloads
$ tar -xzvf otelcol-contrib_0.84.0_darwin_arm64.tar.gz
$ ./otelcol-contrib -v

otelcol-contrib version 0.84.0

NOTE: If you are running on a Mac, you might get the following warning. > The latest Mac OS checks will warn you whenever you try to execute software that has not explicitly gone through Apple's review process (most open-source software falls into this camp). You can bypass this warning by following the instructions here

Create a collector config

The OTEL configuration tells the collector how to manage logs. In our case, we want to receive logs via OLTP and export it to Loki using the lokiexporter

• otel.yaml

receivers:
  otlp/app:
    protocols:
      grpc:
      http:
exporters:
  loki:
    endpoint: "${env:LOGS_ENDPOINT}"
    auth:
      authenticator: basicauth/logs
extensions:
  basicauth/logs:
    client_auth:
      username: "${env:LOGS_USERNAME}"
      password: "${env:LOGS_PWD}"
service:
  extensions: [basicauth/logs]
  pipelines:
    logs/gc:
      receivers:
        - otlp/app
      exporters: [loki]

Note that we use the basicauth extension to authenticate access to our Loki instance. The OTEL collector can make use of environment variables inside the config using the ${env:KEY} syntax.

Get Loki configuration details

NOTE: The following is written with Grafana Cloud in mind and the Loki instance that comes with the free account. If you are self-hosting, the same configuration still applies but your authentication details will differ based on your setup.

You can find the config values for your endpoints by going to https://grafana.com/orgs/{YOUR_ORG} and clicking Details on the Loki data source. You might need to create an authentication token if one does not already exist. You can find further instructions for authenticating with grafana cloud here

Start the collector

For our demo app, we are going to use env files and dotenv to simplify passing configuration details to the collector.

First, install dotenv and create a .env file

npm i -g dotenv
# make sure to not commit your .env file
echo ".env" >> .gitignore
touch .env

Fill the .env file with the relevant configuration values

LOGS_ENDPOINT=
LOGS_USERNAME=
LOGS_PWD=

Start the collector in a separate terminal

# to simplify instructions, we move the collector to the same directory as our application
$ mv /path/to/otelcol-contrib .

$ npx dotenv -- ./otelcol-contrib --config otel.yaml

2023-09-10T11:15:48.834-0700    info    service/telemetry.go:84 Setting up own telemetry...
...
2023-09-10T11:15:48.835-0700    info    service/service.go:161  Everything is ready. Begin running and processing data.

At this point, you should have three terminals carrying out the following functions:

• running your application

• running the collector

• making curl requests to the application

Update the App to Export Logs

We now need to update our app to send logs to the OLTP endpoint (instead of STDOUT).

First, add the required OTEL package to export logs

npm i @opentelemetry/exporter-logs-otlp-proto

Next, update the SDK initialization logic to use the OLTPLogsExporter

• instrumentation.js

diff --git a/instrumentation.js b/instrumentation.js
index 3f76659..663129d 100644
--- a/instrumentation.js
+++ b/instrumentation.js
@@ -2,10 +2,12 @@
 const { NodeSDK } = require('@opentelemetry/sdk-node');
 const {
   SimpleLogRecordProcessor,
-  ConsoleLogRecordExporter,
 } = require('@opentelemetry/sdk-logs');
+const {
+  OTLPLogsExporter
+} = require('@opentelemetry/exporter-logs-otlp-proto');
 
 const sdk = new NodeSDK({
-  logRecordProcessor: new SimpleLogRecordProcessor(new ConsoleLogRecordExporter()),
+  logRecordProcessor: new SimpleLogRecordProcessor(new OTLPLogsExporter()),
 });
 sdk.start();

Finally, restart the app

# cancel the running server 
$ CTRL-C 

# restart
$ node -r ./instrumentation.js app.js

Now send some requests to the app endpoint. You'll notice that logs no longer show up in the console.

Query for Logs

If everything is instrumented correctly, logs should now be appearing in Loki. To find them, login to Grafana and go to the Explore panel of the Grafana console

Choose the Loki data source

Run the following query

{exporter="OTLP"}

NOTE: The search syntax we used is LogQL. It is similar to PromQL (for good and bad...) and lets you search logs in Loki.

You should get the following output

The lokiexporter automatically attaches three labels by default to all outgoing logs: exporter (always set to OTLP), job (set to service.name), and level (log level).

NOTE: Loki differs from traditional solutions like elasticsearch in that it only indexes labels instead of the entire log line.

Our initial query tells Loki to fetch all logs that have the exporter=OLTP label. By default, the lokiexporter does not turn any other fields into labels.

Add Labels

If you inspect the JSON payload, you'll notice that the log message has attributes and resources

{
  "body": "exiting /rolldice",
  "attributes": {
    "roll": 3
  },
  "resources": {
    ...
    "service.name": "unknown_service:node",
    "telemetry.sdk.language": "nodejs",
  },
  ...
}

In OTEL, a resource is metadata about the entity producing telemetry (eg. the service name). An attribute is all other metadata (eg. the roll).

The lokiexporter looks for the special attribute key loki.attribute.labels to determine whether additional attributes should be converted to labels. The resource attribute key loki.resource.labels serves the same purpose for resources.

You can add additional metadata to the log payload to tell Loki to treat certain fields as labels. While we could do this in code, it's much more efficient to do this as a processor in the collector. This allows us to automatically add the metadata to all log telemetry without changing the code.

To add custom attributes in the collector, use the attribute processor. Use the resource processor to do the same for resource metadata. We will now use these processors to add roll and service.name as Loki labels.

diff --git a/otel.yaml b/otel.yaml
index badf962..e74378c 100644
--- a/otel.yaml
+++ b/otel.yaml
@@ -3,6 +3,17 @@ receivers:
     protocols:
       grpc:
       http:
+processors:
+  resource/loki:
+    attributes:
+      - action: insert
+        key: loki.resource.labels
+        value: service.name
+  attributes/loki:
+    actions:
+      - action: insert
+        key: loki.attribute.labels
+        value: roll
@@ -19,4 +30,5 @@ service:
  pipelines:
     logs/gc:
       receivers:
         - otlp/app
+      processors: [resource/loki, attributes/loki]
       exporters: [loki]

Now restart the collector. Like restarting the server, this involves sending a terminating signal to the collector and restarting

# in the terminal where the collector is running
CTRL-C

# restart the collector
$ npx dotenv -- ./otelcol-contrib --config otel.yaml

NOTE: Since all the changes are on the collector side, we do not need to restart the application or change the application code for the changes to take effect.

Now send a few more requests to the application. Query for logs in Loki again and you should see the following result:

roll and service.name are now both labels.

NOTE: Loki does not allow for . in the label name - the lokiexporter automatically translates . into _

You can now directly query the new labels. For example, querying by roll value

{roll="4"}

Use Semantic Conventions

If you look closely at the JSON log payload, you might have noticed that the service name is currently not helpful: unknown_service:node. This is because we have not yet set the name in our application.

The service name can be set when the SDK is initialized. It serves as the primary identifier of the service being instrumented. The service name key and its description are specified by the OTEL Semantic Convetions. These are a set of common values used to describe metadata associated with telemetry.

You can access semantic convention keys by using the language-specific helper packages.

npm i @opentelemetry/semantic-conventions @opentelemetry/resources 

• instrumentation.js

diff --git a/instrumentation.js b/instrumentation.js
index 663129d..4e2d651 100644
--- a/instrumentation.js
+++ b/instrumentation.js
@@ -6,8 +6,19 @@ const {
 const {
   OTLPLogsExporter
 } = require('@opentelemetry/exporter-logs-otlp-proto');
+const { Resource } = require('@opentelemetry/resources');
+const {
+  SemanticResourceAttributes,
+} = require('@opentelemetry/semantic-conventions');
 
+const resource = Resource.default().merge(
+  new Resource({
+    [SemanticResourceAttributes.SERVICE_NAME]: "otel-node-demo",
+    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.0",
+  }),
+);
 const sdk = new NodeSDK({
+  resource,
   logRecordProcessor: new SimpleLogRecordProcessor(new OTLPLogsExporter()),
 });
 sdk.start();

Since we want to index the service version, we are also going to update our collector configuration to convert it to a Loki label.

• otel.yaml

diff --git a/otel.yaml b/otel.yaml
index badf962..e74378c 100644
--- a/otel.yaml
+++ b/otel.yaml
processors:
  resource/loki:
    attributes:
      - action: insert
        key: loki.resource.labels
-        value: service.name
+        value: service.name, service.version

Because we've changed both the collector and the instrumentation code, we need to restart both components for the changes to take effect.

After restarting and sending a few requests, query again in Loki to see both the service name and version show up in the results

Add Auto Instrumentation

By adding OTEL traces to our application, we get trace information in our logs for free.

First, install the auto instrumentation package. This will automatically inject traces into your application at run time.

npm i @opentelemetry/auto-instrumentations-node

Next, initialize the OTEL SDK with the auto instrumentations.

• instrumentation.js

diff --git a/instrumentation.js b/instrumentation.js
index 4e2d651..6a0f794 100644
--- a/instrumentation.js
+++ b/instrumentation.js
@@ -10,15 +10,19 @@ const { Resource } = require('@opentelemetry/resources');
 const {
   SemanticResourceAttributes,
 } = require('@opentelemetry/semantic-conventions');
+const {
+  getNodeAutoInstrumentations,
+} = require('@opentelemetry/auto-instrumentations-node');
 
 const resource = Resource.default().merge(
   new Resource({
     [SemanticResourceAttributes.SERVICE_NAME]: "otel-node-demo",
-    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.0",
+    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.1",
   }),
 );
 const sdk = new NodeSDK({
   resource,
   logRecordProcessor: new SimpleLogRecordProcessor(new OTLPLogsExporter()),
+  instrumentations: [getNodeAutoInstrumentations()],
 });
 sdk.start();

Now, restart the application, make a few requests, and run a new query in Loki.

You should see the following fields in the log payload. These values were added automatically by the trace.

{
  "body": "exiting /rolldice",
  "traceid": "b94fec1be8a7c4ac52b1d633b18afc17",
  "spanid": "acf2dd52184b791e",
  ...
}

Even though we're not using traces in our app, having the traceid and spanid as part of the log metadata makes the logs more useful as you can query those ids to get what are essentially trace events.

If we took the extra step of actually exporting traces, we would be able to jump directly to the trace from the log (aka the big blue button from the diagram below). Unfortunately, we won't get any results right now since we haven't done that.

Export Traces

Let's export traces to see what results would look like. First, install the tracing exporter package.

npm i @opentelemetry/exporter-trace-otlp-proto

Next, update the OTEL configuration to send traces to Tempo. Tempo is an open-source distributed tracing backend that is also available in Grafana Cloud. Don't forget to also update your configuration in .env to pass in the TRACE_* configuration.

• otel.yaml

diff --git a/otel.yaml b/otel.yaml
index 167a6b2..5ac7201 100644
--- a/otel.yaml
+++ b/otel.yaml
  exporters:
+  otlp/traces:
+    endpoint: ${env:TRACES_ENDPOINT}
+    tls:
+      insecure: false
+    auth:
+      authenticator: basicauth/traces
 extensions:
+  basicauth/traces:
+    client_auth:
+      username: "${env:TRACES_USERNAME}"
+      password: "${env:TRACES_PWD}"
 service:
   extensions: [basicauth/logs]
   pipelines:
+    traces/gc:
+      receivers:
+        - otlp/app
+      exporters: [otlp/traces]

NOTE: Tempo has built-in OTLP support and thus works with the builtin OLTP exporter

Finally, update the app to export traces

• app.js

diff --git a/instrumentation.js b/instrumentation.js
index 6a0f794..2265a4d 100644
--- a/instrumentation.js
+++ b/instrumentation.js
@@ -13,16 +13,20 @@ const {
const {
 getNodeAutoInstrumentations,
} = require('@opentelemetry/auto-instrumentations-node');
+const {
+  OTLPTraceExporter,
+} = require('@opentelemetry/exporter-trace-otlp-proto');

const resource = Resource.default().merge(
 new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: "otel-node-demo",
-    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.1",
+    [SemanticResourceAttributes.SERVICE_VERSION]: "0.1.2",
 }),
);
const sdk = new NodeSDK({
 resource,
 logRecordProcessor: new SimpleLogRecordProcessor(new OTLPLogsExporter()),
+  traceExporter: new OTLPTraceExporter(),
 instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

With the changes in place, restart both the collector and the app. Send a request and query for logs in Loki. When the log results show up, you should now be able to jump from log to the corresponding trace.

Note that this also works in reverse.

Final Thoughts

It took three years from the inception of OTEL for the logs API to finally hit GA. While SDK maturity currently varies by language, it is now more usable and can bring big benefits.

In some sense, logging maturity in OTEL has a greater impact on observability than traces. As useful as traces are, most organizations still don't use them. All organizations use logs. By using logs via OTEL, you can even get many of the benefits of tracing (eg. automatic traceid and spanid) without having to store traces.

And for organizations that make use of all three pillars, OTEL breaks down the barriers between signals and enables autocorrelation between traces, logs, and metrics.

All this to say - logs are great. OTEL can make them better.

OpenTelemetry (OTEL) started in 2019 when the two dominant open-source projects around tracing, OpenTracing and OpenCencus, merged to form OpenTelemetry. The scope of the combined project expanded beyond tracing to cover all of observability. OTEL's mission is to help organizations deliver high-quality, ubiquitous, and portable telemetry.

Four years later, OTEL is making good on its promise. It has to date provided a stable standard for the three pillars of observability (metrics, logs, and traces), a collector that can receive, process, and export telemetry in any environment, and SDKs to instrument code in all major languages. It has also continued to expand its scope and introduced additional standards around semantic conventions and agent management.

Today, OTEL is the second most active project in the CNCF, behind only Kubernetes in popularity. Its contributors are spread across all major observability vendors and its protocol has near-universal adoption among observability providers.

In this post, we'll go over where OTEL is today, provide a summary of the various projects that fall under its domain, and talk about where it's going next.

OTEL - What's in a name?

OTEL consists mainly of two types of projects: specifications and implementations.

Specifications are the foundation of OTEL. They describe how telemetry should be captured, collected, processed, and exported. Specifications tend to be vendor-focused as they define the common standard that vendors have to implement to be compatible with OTEL.

Implementations are the specific client libraries and tooling that are used to handle telemetry data. Implementations tend to be end-user-focused as they consist of the parts that end-users use to instrument their code.

A project's stability is typically marked on a signal-by-signal basis. Each signal is a specific type of telemetry data - in OTEL, this means metrics, logs, and traces.

OTEL today consists of the following sub-projects:

• OpenTelemetry Specification (specification)

• OpenTelemetry SDKs (implementation)

• OpenTelemetry Protocol (specification)

• OpenTelemetry Collector (implementation)

• Open Agent Management Protocol (specification)

• OpenTelemetry Semantic Conventions (specification)

OpenTelemetry Specification (1.24.0)

The Open Telemetry specification is the foundation of OTEL - it provides the APIs, SDKs, and data models that all other OTEL standards are derived from.

A brief timeline of its development:

• 2020-09: Tracing marked stable

• 2021-11: Metrics marked stable

• 2023-04: Logs marked stable

In terms of maturity, the OTEL spec is now stable for all signals. Because the logging signal was not deemed stable until earlier this year, many OTEL SDKs still do not support logging at this time.

OpenTelemetry SDKs (Mixed)

The OTEL SDKs provide client-side instrumentation based on the OTEL spec. For each language-specific SDK, there is a separate maturity level for each signal.

Current maturity of each SDK.

Auto Instrumentation

Depending on the programming language, some SDKs also support automatic instrumentation. This is when the SDK automatically injects signals, mainly traces, into your application without needing to manually instrument code.

Auto instrumentation doesn't work for compiled languages like go and rust. That said, you can still get automatic trace injection by going outside of the SDK and using eBPF and/or service mesh based tooling.

OpenTelemetry Protocol (OTLP - 1.0)

OTLP describes a common wire protocol for delivering observability data. There are two blessed transports in OTLP: http with protocol buffers and gRPC.

This spec is considered stable and can be implemented on any service that receives, processes, or exports OTEL data. The spec is implemented by the OpenTelemetry collector and also by agents of observability vendors like Grafana and Datadog.

OpenTelemetry Collector (0.83.0)

The OTEL collector is a vendor-agnostic agent that collects, transforms, and sends observability data.

The collector consists of the following components:

• receivers: push/pull data from different sources

• processors: transform/filter/enrich/derive data in flight

• exporters: send data to downstream destinations

• connectors: connectors are both a receiver and exporter and enable multiple pipelines to be stitched together

• pipelines: a chain of receiver, zero or more processors, and exporters

• extensions: components that provide additional capabilities outside of handling telemetry data (eg. basic auth, health checks, etc)

Together, the various components act as an observability pipeline, allowing you to collect telemetry data from any source, process it in flight, and send it to any destination.

The OTEL collector consists of two projects, otel-collector and otel-collector-contrib. otel-collector contains only the core components of the collector, chiefly logic strictly related to processing OLTP data.

otel-collector-contrib is a kitchen sink of every integration you can imagine, with exporters and receivers for most observability providers present. At the time of this writing, this includes 91 receivers, 48 exporters, and 24 processors.

End users are advised to use the OpenTelemetry Collector Builder to create a custom build of otel-collector-contrib with only components that they need.

In addition to otel-collector-contrib, vendors like AWS and Splunk provide their own custom distributions of OTEL.

Open Agent Management Protocol (OpAMP - Beta)

OpAMP is a network protocol for the remote management of agents. This is a recent addition to OTEL, introduced in 2022, and provides a vendor-agnostic standard for controlling fleets of agents. These agents could be instances of the otel-collector or vendor-specific agents that implement OpAMP.

With OpAMP, you can enable features like dynamic configuration deployments, agent updates, and credential management.

There is currently a work-in-progress go implementation of the OpAMP spec.

NOTE: If you're interested in managing OTEL agents via OpAMP - this is what my startup is working on.

OpenTelemetry Semantic Conventions (1.21.0)

The OTEL Semantic Conventions define a common set of attributes for observability data. They cover a wide range of areas including cloud resources, databases, exceptions, and systems.

The semantic conventions are used by the OTEL SDKs and automatically applied for SDKs that support auto instrumentation. Common semantics enables the ability to correlate between different signals and is something I'm especially excited by, the reason for which can (and probably will) be the subject of an entire blog post.

Etc

OTEL is a massive project. While we covered the flagship projects, there are additional parts of OTEL that deserve a notable mention:

• OpenTelemetry Transformation Language (OTTL): a common transformation language for telemetry that is both performant and flexible - currently designed and implemented in otel-collector-contrib

• OTEL Demo: A micro-service-based shopping site that demonstrates a majority of features and language SDKs of OTEL

Final Thoughts

OTEL started as a merger between competing tracing specifications and evolved to become an industry standard for observability. The last four years laid the groundwork for building a common foundation across vendors and tools. The next four will reveal the fruits of that labor.

Subscribe now

When people speak of observability (assuming they talk about it at all), they tend to focus on the "backend" - aka systems responsible for ingesting, storing, and querying observability data (metrics, logs, and traces). But this is only half the story.

For observability data to make it into these systems, it first needs to be collected, processed, and exported from the infrastructure and services that generate the data to where it will be used. This "front end" of observability is what is known as "first-mile observability". Solutions in this space have grown in scope in recent years due to the increase in volume (and cost) associated with modern observability data.

In this post, we'll go over first-mile observability, what it is, and the landscape of offerings that exist in this space today.

A Series of Pipes

At its core, first-mile observability represents systems that facilitate the collection, processing, and exporting of observability data. They're centered around the following components:

• receivers: push/pull data from different sources

• processors: transform/filter/enrich/derive data in flight

• exporters: send data to downstream destinations

NOTE: Different implementations have different names for these components. I'm using the terminology adopted by the Open Telemetry (OTEL) collector.

The combination of receivers, zero or more processors, and exporters, come together to create an observability pipeline.

Some examples of common pipelines:

• a pipeline that collects all log data and sends 100% of it for archival to S3 but only ERROR logs to datadog

• a pipeline that adds host and container level information as custom attributes to each data point before sending it upstream

• a pipeline that removes unused attributes and dimensions from observability data

Example pipeline that filters logs sent to datadog while sending everything to S3

First Mile Concepts

Agents

The most common deployment in first-mile observability is that of an agent. Agents tend to run on the same system that they are collecting data from. They are responsible for pulling in data, doing lightweight aggregations, and then sending the data off to a preset destination.

Every observability vendor has a custom agent.

Collectors

A step up from the agent deployment is the collector deployment. Collectors can be deployed either on the same system that they are monitoring or as a standalone deployment. They aggregate observability data from various sources, can apply basic filters, and route data to one or more destinations.

FluentD and Fluent Bit are some of the most popular collector options with wide community adoption and extensive integration solutions.

Pipelines

When a collector grows up, it becomes an observability pipeline. Observability pipelines can be deployed on the same system, as a standalone deployment, or as a cloud endpoint. Observability pipelines can receive data from any source, apply near-arbitrary transformations on the data when it is in flight, and export it to any destination.

Vector is a popular open-source pipeline solution written in Rust that has widespread adoption.

First Mile Landscape

There are many solutions for observability pipelines - we'll go over some of the most popular solutions in both the open source and commercial sides of this space.

Open Source Solutions

In terms of open source, popular solutions include Fluent Bit, Vector, and the Otel Collector

Fluent Bit

• released: 2015

• language: C

• pros: performance, Lua based transformations

• cons: smaller ecosystem (compared to FluentD), harder to extend

Fluent Bit was released in 2015 as a lightweight version of FluentD. It has zero dependencies, is highly performant, and supports arbitrary transformations that can be authored with the Lua programming language. It also comes built-in for certain cloud vendors like AWS.

Vector

• released: 2020

• language: Rust

• pros: performance, custom transformation language (VRL), acknowledgment and delivery guarantees

• cons: smaller ecosystem, hard to write plugins

Vector was released in 2020 as a modern rust-based observability pipeline. It achieves even higher throughput than Fluent Bit (albeit with higher CPU consumption) and is easy to install and integrate. Users can transform data using the vector remap language - a custom DSL that is designed for transforming data while maximizing flexibility, runtime safety, and performance.

OTEL Collector

• released: 2020

• language: Go

• pros: performance, large ecosystem, industry standard, easy to extend

• cons: mixed stability

The OTEL collector was released in 2020 as part of the Open Telemetry (OTEL) project. OTEL is a CNCF initiative to provide a vendor-neutral standard for cloud-native observability. The OTEL collector can be deployed and used as either an agent, collector, or pipeline depending on how it's configured.

The OTEL collector is available in many different flavors. The core collector provides a stable core of built-in receivers, processors, and exporters to facilitate OTEL formatted data. The OTEL Contrib Distro contains the kitchen sink of integrations into every vendor and observability service under the sun. Vendors like Splunk and AWS also offer their own curated OTEL Distros that are specifically curated for their particular platform.

Users can transform data using built-in processors or use the OpenTelemetry Transformation Language (OTTL). This is a SQL-like declarative DSL optimized for transforming observability data. Note that OTTL is under active development.

Commercial Solutions

There are also multiple commercial offerings available in the observability pipeline space. Popular vendors include Cribl, Calyptia, Datadog, and Mezmo.

Cribl

• founded: 2017

Cribl was started by a group of ex-splunk folks. Their original claim to fame was saving money on your Splunk bill. Cribl is known as the first major commercial offering in the observability pipeline space. They are an easy choice for the enterprise and companies that are heavily reliant on Splunk.

Calyptia

• founded: 2020

Calyptia was founded by the authors of FluentD and FluentBit. They provide an observability pipeline solution built on top of said products. They are a good choice if you already use these technologies or operate on platforms that integrate natively with them.

Datadog

• founded: 2010, acquired Vector in 2021

Datadog has dominated the modern observability landscape by positioning itself as the single pane of (expensive) glass for all observability needs. They acquired the company behind Vector in 2021 and have built a managed observability pipelines solution around it. They are a good choice if you're already using Vector and want native support for datadog.

Mezmo (formerly LogDNA)

• founded: 2015, rebranded to Mezmo in 2022

Mezmo started as LogDNA with an exclusive focus on providing a logging pipeline. It has since expanded its scope and also rebranded itself to be a general observability pipeline solution. Mezmo is easy to get started and has a slick UI for adding filters and transformation. They have a proprietary agent for their logging pipeline but their observability pipeline can ingest data from a variety of different sources, including fluentd, fluent bit, and OTEL.

Nimbus

• founded: 2023

Nimbus was started in 2023 by the author. It is a fully managed observability pipeline that helps organizations reduce data volume by 60% without losing signal. Nimbus analyzes your traffic and automatically recommends optimizations that reduce and aggregate data.

Choosing the Right Solution

There are many options for first-mile observability. The easiest and most common choice is to do nothing and use whatever agent(s) comes bundled with your vendor/solution of choice.

If that no longer suffices, some considerations for choosing an observability pipeline below:

• For something performant that just works, consider Vector. Use DataDog for a managed version.

• For integrating with the widest set of existing tools, consider FluentD and Fluent Bit. Use Calyptia for a managed version.

• For enterprise and Splunk buildouts, consider Cribl. There is no open-source equivalent though Cribl does offer a free tier.

• For a good mix of everything as well as future-proofing with Open Telemetry, use the OTEL collector.

Final Thoughts

In 2023, not only is observability data growing at a dizzying rate, but so are vendors that help you manage it. Whereas the "backend" space has a vast selection of options for both incumbents and newcomers, the "frontend" space did not develop commercially until 2017 with the launch of Cribl. Now six years later, observability pipelines are only now starting to gain traction. I'm personally excited by the potential of OTEL - its CNCF status, standards-based implementation, and transformation language position it to be a dominant leader in this space.

An observability platform is an end-to-end system that helps organizations understand the health of their applications and services. This understanding comes from the three pillars of modern observability: metrics (time series data), logs (text-based data), and traces (request data with associated baggage/metadata).

The challenge of modern-day observability is scale - instead of a single host running a LAMP stack generating a few megabytes of observability data per day, we now have Kubernetes clusters with thousands of services generating gigabytes of observability data every hour.

The challenge of collecting, ingesting, storing, and querying observability data at scale is what modern observability platforms are designed to tackle. Depending on the underlying architecture, the cost of running these platforms can vary by a factor of over 100x. This post covers different types of architectures and solutions that implement them.

The Observability Pipeline

Before diving into architecture, an overview of concepts. There are four distinct phases when it comes to making use of observability data:

• collection: observability data is received at the edge (usually in the form of an agent running on a host)

• ingestion: observability data is processed at the destination (usually involves batching, compression, and additional transformations to get the data in the optimal shape for storage)

• storage: observability data is retained (usually involves indexing)

• query: observability data is looked up (usually involves translating a query to a GET/LIST request to the underlying storage system)

Observability Architectures

Distinct and Separate Components

As observability evolved from checking syslog to dedicated systems devoted to monitoring, early platforms build highly specialized services for each of the observability pillars. This meant that metrics, logs, and traces were processed by completely independent systems with distinct collection, storage, and query layers.

In the open-source world, this might take the form of running Prometheus, Elasticsearch, and Jaeger. The pipeline for each service is listed below.

Prometheus for metrics

collect (prometheus scraper) -> ingest(prometheus) -> store (prometheus) -> query (prometheus)

Elasticsearch for logs

collect (logstash) -> ingest (elasticsearch) -> store (elasticsearch) -> query (elasticsearch)

Jaeger for traces

collect (jaeger collector) -> ingest (jaeger) -> store (cassandra) -> query (jaeger)

Observability vendors that were founded during this time (eg. splunk, new relic, datadog) were likely built on a similar foundation (of distinct and separate services for each pillar).

NOTE: While the specifics of the implementation that vendors use are not public knowledge, their pricing model, both the dollar value as well as the dimensions that they charge for (eg. the amount of data indexed) suggest they use this architecture

Unified Collection

As observability grew in scope (and vendors in abundance), standards started emerging. This culminated with Open Telemetry (OTEL) being released in 2019 from the merger of the two big projects in this space: OpenTracing and OpenCensus.

In 2023, this standard has (mostly) come into maturity - they provide a vendor-neutral spec and implementation to collect metrics, logs, and traces from any source and send them to any destination.

All observability vendors today have support for OTEL.

Unified Storage

As observability data grew more voluminous, folks started to realize that the approach of using high-performant databases to store and index all this data did not scale.

Instead of indexing everything upfront in an expensive database, you can get order-of-magnitude improvements in cost per byte by only doing a partial index and storing the data in cloud object storage solutions like S3 (usually with gzip + parquet).

In the open-source world, you can find this architecture in the following solutions:

• Prometheus with Cortex/Mimir backend for metrics

• Loki for logs

• Tempo for traces

Observability vendors that use an object storage backend: logz, grafana cloud, and axiom.

NOTE: Clickhouse is a high-performance open-source columnar database that is also a popular solution for storing metrics, logs, and traces. One downside with clickhouse is that it couples the storage layer with the query layer which means that fundamental decoupling of the two (which we'll cover later in this article) is infeasible in clickhouse-based architectures.

Unified Ingestion

When all observability data is stored with unified storage, it becomes possible to also radically simplify its ingestion.

Ingestion is difficult at scale because of data volumes, throughput, and sudden changes in traffic patterns. Instead of standing up separate services to ingest data coming in for each pillar, you can consolidate everything using a streaming platform like Kafka. These platforms are designed to ingest data in real-time at scale and also allow real time enrichment and transformation of the data as it comes in.

I don't know of any open-source observability projects that use this architecture. Presumably, this is because you only start accruing benefits once you unify all three pillars with a common storage backend - otherwise, it's more efficient to build out a dedicated ingestion mechanism than taking the overhead of standing up something like Kafka

The observability vendor Coralogix employs a unified streaming-based architecture that does something similar.

Unified Querying

When all observability data is stored with unified storage, it becomes possible (and often necessary) to simplify the querying.

Querying terabytes of data is expensive when the data is indexed and hard to do at an acceptable latency when it is not. This is where massively parallel processing (MPP) techniques come into play - by spinning up many processors to independently process data (using tools like spark and Trino) and coupled with a metadata store like a hive or Apache Iceberg, it becomes possible to process terabytes of "un-indexed" data in seconds.

Like with ingestion, I don't know of any open-source observability platforms that use this architecture. The reason for this is similar to that of ingestion - unless you build out an observability platform that is made from day one to support all the pillars with unified storage, it's not worth the overhead of standing up something like Spark or Trino.

The observability vendor Observe employs a unified query architecture using Snowflake as its MPP engine.

Final Thoughts

There's a lot of observability data in the world and many platforms that enable you to make use of it. This is all at a price - the major driver of that price is the underlying architecture of the observability platform (and the margins that vendors want to make).

The unification of the storage layer enabled the ability to scale observability platforms while reducing costs by an order of magnitude. The unification of the ingestion layer and the querying layer has the potential to bring about similar benefits.

There's an opportunity today to create a massively more scalable and affordable observability platform by unifying every layer of the observability pipeline. In a prior post, we show how doing this for logs can bring down cost by over 95% when compared to datadog.

This sort of unified architecture effectively commodifies observability at scale. This is a good thing as the amount of data that is observed continues to grow - at the rate we're going, even enterprise companies are starting to look for alternative solutions due to costs.

All this said, being able to have the data on hand is just table stakes - the real value is being able to translate that information into insights and business outcomes. The sooner we can stop worrying about cost, the sooner we can get back to doing just that.

Athena released provisioned capacity in April 2023. This has been one of the most asked-for features ever since Athena launched in 2016. In this post, we'll go over what provisioned capacity is, why it matters, and how it holds up in practice.

As a refresher, Athena is a serverless analytics offering by AWS that "provides a simplified, flexible way to analyze petabytes of data where it lives". Underneath the hood, Athena runs on presto/trino.

Athena is pure serverless - it has zero setup and you only get billed when you query. Billing for Athena is based on the data scanned and is charged at $5/TB or $0.005/GB (for the keen-eyed, this is the same price as cloudwatch log insights).

Note that there are additional fees due to GET/PUT calls to S3 and AWS Glue - these costs are generally negligible and don't meaningfully impact the query cost (though they'll frustrate your accountant).

Athena runs on shared infrastructure and has the concept of a queue time - depending on the load on the Athena fleet in a particular region, you might have to wait before your query gets to run. Most of the time, queue time is under 200ms but for the p99 case, this can go past a minute.

Query execution itself is quite fast, especially on the latest Athena engine . On a well-tuned dataset, Athena can scan +7GB/s and finish in single-digit seconds for simple queries, making it suitable for real-time analysis.

While there is both extensive documentation and controls for tuning both your dataset and your queries for optimal performance, there has not been, until this year, a way of directly controlling your time in the queue.

Athena Provisioned Capacity

Athena provisioned capacity gives customers a way of reserving capacity on the Athena fleet for exclusive use. By reserving sufficient capacity, you can eliminate queuing and also have predictable billing.

When you provision capacity, you request capacity in Data Processing Units (DPUs). A single DPU is equivalent to 4 vCPUs and 16GB of memory. It is charged at $0.30/DPU.

You have to specify at least 8 DPUs when making a reservation and you need to book it for at least 8h. This comes out to $7.20/hour or $57.60 minimum per use. Once you make a reservation, you are no longer charged for the amount of data scanned but purely for the DPUs that you've reserved.

After 8h of use, DPUs are charged on a per-minute basis. You can cancel a reservation at any time but if you cancel early, you will still be charged for a minimum of 8h of use.

Making a Reservation

Making a reservation is simple and can be done either via the console or the API. Note that reservations can not go through depending on capacity as well as take up to 30 minutes to be ready. That said, in testing, reservations were successful in under a minute.

Once a reservation is made, you need to associate it with one or more workgroups. A workgroup is an Athena concept that encapsulates the Athena engine version, capabilities, and limits on querying for users of the given workgroup.

After completing the association, any queries you make via a workgroup that has a capacity reservation will use the dedicated capacity instead of the on-demand capacity of the Athena fleet.

Using Provisioned Capacity

The good news is that provisioned capacity does eliminate unexpected queue spikes - because of dedicated capacity, you are guaranteed execution (as long as there is capacity).

This comes however with the caveat that average queue time gets worse.

The above image shows observed queue times when running in an on-demand workgroup vs a provisioned workgroup. The average time spent in the queue is ~158ms for on-demand and ~331ms for provisioned (more than 2x). The difference of an extra ~150ms per query vs the variance introduced by an additional 6000ms during a queue spike is generally worth the tradeoff - nevertheless, if you're running lots of short queries, this is something to be aware of.

What is not great about provisioned capacity is that your time in the queue can grow unbounded (for as long as you have an active reservation). This happens if you do not have enough DPUs provisioned. Any workgroup that uses provisioned capacity is limited to only using provisioned capacity as long as the reservation is in place. This means if your in-flight queries are consuming all the DPUs, any additional queries will be queued until capacity becomes available.

AWS Lambda is another serverless offering with a provisioned capacity model. Unlike the Athena model, Lambda's provisioned concurrency allocates pre-initialized environments based on the reservation and falls back to on-demand when it is exhausted. It's an approach that I wish Athena would have followed - falling back to on-demand is a much better user experience than putting a hard stop on queries.

Effectively, provisioned capacity turns Athena from a serverless analytics engine to managed infrastructure that you have to plan capacity around. What makes this worse is the lack of visibility when it comes to mapping DPUs to workloads.

While AWS has a table that provides guidance on how many DPUs to reserve based on the expected number of concurrent queries, this is not enough information to do capacity planning.

This is because DPU requirement varies based on data size, storage format, query construction, and other factors, not just concurrency. As an example, the common-crawl dataset has partitions that exceed 200GB. Running the following select statement requires a minimum of 124 DPUs (this translates to 1984GB of RAM - why is this much RAM needed to scan 200GB of data? I can't tell you why and neither can Athena 🤷‍♂️).

select url from ccindex
WHERE crawl = 'CC-MAIN-2023-06'
AND subset = 'warc' AND content_digest = '3WCXFWELSS4SO3MLCODIDHS2KLT4SHPZ'

Right-sizing how many DPUs you need is a manual guess and check - the current AWS recommendation is to reduce DPUs and see if that causes query failures or increases queue times.

Final Thoughts

I like Athena. It is one of my favorite AWS services and the first thing I reach for when needing to analyze data in S3. The team has done a tremendous job in boosting engine performance and building out a wide feature set. It even works across regions, accounts, and KMS-encrypted objects. The issue of unpredictable queue spikes is one of the few remaining reservations I have about the service.

Provisioned capacity is the first direct solution to this problem. Unfortunately, by using it, you also give up most of the benefits of Athena being serverless. With provisioned capacity, you now have to plan for capacity upfront, you pay even when you're not using the service, and you hit a hard stop if you exceed the set capacity.

I do not doubt that there exists a set of customers that will be well served with this feature - mainly those with consistent workloads that take place over eight hours and place a premium on predictable pricing. For everyone else, you're probably better served continuing with on-demand.

If you're using AWS, there's a good chance (>95%) you're using Cloudwatch Logs (CL). It is the defacto logging solution from AWS - CL lets you collect, store, and query log files from anywhere with little to no upfront work.

If you're using CL, there's also a good chance you're paying for more than you need.

Why Cloudwatch Logs

First, lets talk about why you might be using CL.

CL comes ready out of the box. You might not even be aware that you're using it as it integrates directly with 30+ AWS services (API Gateway, Lambda, ECS, etc).

CL can also be extended into on-prem environments using the unified cloudwatch agent - you can use the agent to collect both metrics and logs from the host machine.

CL covers the basic use cases for logging - querying (via Cloudwatch Log Insight) and tailing (via live tail). It also lets you use filters to create metrics from logs and add subscribers that can forward your logs to other destinations.

CL's most compelling (and expensive) feature is that it can do all of this in "near real time" - logs usually show up ~5s-20s after publication (CL publishes times for when logs are emitted vs ingested so you can actually do the calculation yourself).

Why not Cloudwatch Logs

CL might not be for you if you don't need "near real time" logs and the cost of operation is too high.

This is because the real time nature of CL comes at a premium - the ingestion fee per gigabyte ($0.50/GB) is one of the highest in the industry (for comparison, new relic is $0.30/GB, datadog is $0.10/GB, and S3 is free).

CL charges you based on ingestion and retention. Ingestion is $0.50/GB. Retention is $0.03/GB. Ideally, you'd want the costs to be flipped. When retention is expensive, you can reduce costs by shortening you retention policy and archiving older logs to S3. But when ingestion is expensive, your options are limited and generally involve not sending logs at all.

Comparing Cloudwatch Logs with Other Solutions

Lets compare CL to some other logging solutions in terms of cost. Specifically, we'll compare:

• cost of using CL

• cost of using Datadog (industry standard)

• cost of a custom solution using other AWS services

We'll make the following assumptions to baseline our comparison:

• Net new logs ingested per day: 500GB/day

• Average size of a single log event: 256 Bytes

• Retention period for logs: 90 days

• Amount of data scanned per day: 3,500GB (7 days worth of logs)

When calculating log costs, we'll consider the following factors:

• ingestion fee: how much does it cost to import the logs

• retention fee: how much does it cost to store the logs

• querying fee: how much does it cost to search the logs

1. Cloudwatch Logs

• Ingestion: $0.50/gb -> for 500GB/day, this is $250/day.

• Retention: $0.03/gb -> for 500GB/day, this is $15/day

• Querying (via Cloudwatch Log Insight): $0.005/GB scanned -> for 3500GB, this is $17.50/day

• Total: $282.50/day

Note that the retention cost is less than 10% of ingestion cost. CL is expensive primarily because of the high ingest cost.

2. Datadog

• Ingestion: $0.10/GB -> $50/day

• Retention: $1.70/1 million events -> assuming 256B per event, 500GB translates to 1,953,125,000 events or $3,320.31day

• Querying: $0

• Total: $3,370.313/day

Note that Datadog's pricing is flipped when compared to CL - ingestion is cheap but retention is very expensive (~2200% more in this case). Your exact costs will vary depending on the size of your event but in most cases, you'll pay more for retention with Datadog than with any other vendor.

3. Custom Solution using other AWS Services

There's many ways to build a logging platform on top of AWS. For today's comparison, we will use Firehose (managed streaming service), S3 (managed object storage), and Athena (managed query/analytics engine on top of S3). These are all mature serverless offerings provided by AWS that require no ongoing operational effort to run and are also cost effective at scale.

Calculating the cost is a bit more involved and differs per service.

• Ingestion:

  • Firehose: $0.029/GB -> $14.50/day

  • S3: Ingest is free but you do pay $0.005 per 1k PUT request. Assuming you set the buffer at 1MB for Firehose, this works out to 500GB/0.001GB * $0.005/1000 or $2.50/day

  • Total: $14.50/day

• Retention: $0.023/GB for S3 standard storage or $11.50/day

• Querying:

  • Athena: $0.005/GB scanned or $17.50/day

  • S3: Athena issues GET requests to S3 which costs $0.0004 per 1k GET request. Assuming average object size of 1MB, this works out to 3500GB/0.001GB * $0.0004/1000 or $1.40/day

• Total: $47.40/day

Our home grown solution made from combining different AWS services stands out in a few respects compared to the other solutions. Most notably, it is 5x cheaper than CL and 98% cheaper than Datadog for the same amount of logs. More importantly, there is no premium on either ingestion or retention which means that costs stay manageable even as you grow.

One caveat is that our home grown solution has a variable cost when it comes to querying data. How much this varies depends on how your data is stored as Athena charges per GB scanned. This is impacted by your file format (eg. Parquet, Iceberg, etc), indexing, and partition strategies. It is also depends on what kind of queries you run and how often you run them.

The other caveat is that the latency of streaming logs from Firehose to S3 can be on the order of minutes instead of seconds so it might not work as well if you need near real time logs.

Finally, this doesn't have the other nice features that CL has - mainly, metric filters and subscriptions.

Note that depending on your requirements, any and all of these caveats can be addressed. You can eliminate the query cost by using UltraWarm storage with Opensearch. You can get near real time logs, metric filters, and subscriptions by using Kinesis Data Streams instead of Firehose. Each of these solutions have different cost/complexity tradeoffs but will, at scale, still be cheaper than running CL or Datadog.

Costs of Different Logging Solutions

Here's a table that shows the various cost calculations in one place.

Final Thoughts

When you're starting out, everything is cheap (even datadog). You can and should use whatever works best for your team and business.

This article is not advocating that you not use CL - it's a fantastic service (with questionable UX) that takes a lot of toil out of logging. But if you find that log costs are getting you down and especially if you're not taking advantage of the near real time nature of CL, then know that there are better options.

Everything in AWS is about tradeoffs - if you take time to optimize your usage, you will not find a most cost competitive solution at scale (without running your own data centers). The challenge is that it takes a long time to even know what all the options are, never mind actually put them into practice (it doesn't help that AWS releases changes on a daily basis).

In the case of Cloudwatch Logs, understand the tradeoff you're implicitly making and if cost is a concern, make different tradeoffs.

AWS released Amazon CloudWatch Logs Live Tail on June 6th, 2023. The name is on point even if it is a mouthful (though terse by AWS standards) - Live Tail is a service that helps developers follow (tail) cloudwatch logs in real-time. It also layers on additional goodies like dynamic filters and highlights.

The real-time part of the service intrigued me - cloudwatch logs, anecdotally, tend to have a latency between 5s to 20s for log ingestion. Could Live Tail do better? And how does it compare to the existing CLI workflow?

aws logs tail {LogGroupName} --follow --format json

NOTE: aws logs tail is not equivalent to Live Tail and has been available long before

Experiment time!

Setup

To run the test, I set up a serverless stack consisting of API Gateway + Lambda.

The experiment:

1. Use both Live Tail and the AWS CLI to tail the created cloudwatch logs

2. Send an HTTP request to API Gateway

3. Observe when logs showed up in live tail vs the CLI

To measure when logs showed up, I split my window into three panes.

• Pane 1 represented the Live Tail console

• Pane 2 represented the results of running the CLI

• Pane 3 was opened to https://currentmillis.com which showed the current time in milliseconds

I recorded my screen to track when logs showed up in panes 1 and 2. I did this testing across two scenarios - the first from a lambda cold start (this first created a new cloudwatch log stream), the second from a warm start (this appended logs to the existing stream)

You can see a sample run below

I then compared this data to the output of the GetLogEvents API call. This returns events from a particular log group in the following format:

{
  "events": [
    {
      "timestamp": 1687921318305,
      "message": "INIT_START Runtime Version: nodejs:16.v14\tRuntime Version ARN: arn:aws:lambda:us-east-1::runtime:***\n",
      "ingestionTime": 1687921325842,
    },
    ...
}

• timestamp: the time when the event was emitted (from lambda)

• ingestionTime: the time when the event was processed (in cloudwatch logs)

TIP: By subtracting the ingestionTime from the timestamp, you get the cloudwatch log delivery latency.

Observations

The initial call from cold start yielded the following timestamps (in milliseconds since epoch). Timestamps are space-delimited for readability.

call: 16879213 18 305,
inge: 16879213 25 842
tail: 16879213 27 193
foll: 16879213 27 193

The way to read the timestamps:

• call: when API call was made (timestamp field)

• inge: when log was ingested by cloudwatch (ingestionTime)

• tail: when Live Tail started displaying the logs

• foll: when the AWS CLI first started displaying the logs

Based on the initial data, you can see that Live Tail is downstream of cloudwatch logs. Ingestion of logs has a 7s latency. Tailing of the logs has a 2s additional latency. In this case, both tailing and following logs occurred within the same second.

Here is the result of another run, this time from a warm start.

call2: 16879213 38 725 
inge2: 16879213 47 744 
tail2: 16879213 50 096
foll2: 16879213 55 079

This time, there's a 9s ingestion latency followed by a 3s latency for Live Tail. The CLI has another additional 5s latency behind tailing.

I re-ran this experiment a few times and got similar results. Live Tail will stream logs ~2s from when it is first ingested in cloudwatch. Using the CLI, this can vary between 2s to 9s. Note that this is on top of the latency to ingest metrics into cloudwatch logs (usually between 5s-20s).

Implementation

Behind the scenes, I assumed that Live Tail was using websockets. This is hinted at by the pricing model (seconds of active usage) which charged for the number of active connections.

This can be verified by digging into the Chrome network inspector tab. At the start of a Live Tail session, the console makes the following request.

# NOTE: extraneous fields stripped for clarity
curl 'https://us-east-1.console.aws.amazon.com/cloudwatch/CloudWatch/data/auth.SdkSign/20230628033119884-***' \
  -H 'origin: https://us-east-1.console.aws.amazon.com' \
  ...
  --data-raw $'{
    "Service":"logs","Endpoint":"logs.us-east-1.amazonaws.com","HttpMethod":"POST","Region":"us-east-1","Path":"/",
    "Headers":{"X-Amz-User-Agent":"aws-sdk-js/2.1265.0 promise","X-Amz-Requested-Operation":"startLiveTail","Content-Type":"application/x-amz-json-1.1","X-Amz-Target":"Logs_20140328.StartLiveTail",...}'

Note the X-Amz-Target set to Logs_20140328.StartLiveTail. This looked very much like an API call to start a WebSocket connection.

When inspective for websocket connections, you can see said connection

curl 'wss://us-east-1.prod.livetail.cloudwatchlogs.aws.dev/' \
  -H 'Origin: https://us-east-1.console.aws.amazon.com' \
  -H 'Sec-WebSocket-Key: **' \
  -H 'Upgrade: websocket' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: Upgrade' \
  -H 'Sec-WebSocket-Version: 13' \
  -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits' \

Presumably, there is a real-time service in cloudwatch logs that will push new events through the Live Tail websocket as soon as it is ingested.

In this regard, Live Tail is unlike most AWS services. AWS has a reputation for being an API-first platform (Route53 launched with only API support back in 2010). It feels strange to see a new feature be launched that is console first - at the time of this writing, there is no way (that I know of) to integrate Live Tail via API.

Parting Thoughts

Live Tail is a service that works as advertised. It tails your cloudwatch logs in real-time with the caveat that it is subject to the same ingestion latency that is inherent in being downstream of cloudwatch logs.

While the CLI-based solution has existed for ages, Live Tail brings this to the AWS console with additional features (eg. the ability to tail multiple log groups, filtering, and syntax highlighting). The difference between Live Tail and the tail cli command is that one is push-based and the other is pull-based. Live Tail can deliver logs from cloudwatch with consistent single-second latencies whereas, via the CLI, it is variable.

If you're already using the CLI, there's not much incentive to switch (unless you need the ability to tail multiple log groups at once). If you rely on the console for log diving, then Live Tail will be a welcome addition to your tool set.

Do note that Live Tail is not free. It has a heart palpitation-inducing per-second pricing model ($0.01 per minute) that is offset, in part, by the 1,800 free minutes per month free tier (that's 1h per day).

All in all, the addition of Live Tail fills the last gap in the cloudwatch suite of observability tooling. Just remember to not leave it running for too long!

On April 27, 2023, AWS rolled out a long-anticipated change to Amazon S3. This change blocked public access and custom ACL rules, by default, for all newly created buckets on S3.

Bad S3 bucket permissions are among the most widely exploited "security incidents" in the cloud. This occurs when someone creates a bucket for private data (eg. personal customer information) but accidentally makes that bucket publicly available. By blocking public access, AWS makes this sort of misconfiguration impossible (without explicitly disabling the block policy).

But no good deed goes unpunished - for every step forward AWS makes security-wise, it feels that they go two steps back on the developer experience (DX). While public buckets can no longer be created by accident, they are also now much more difficult to create on purpose.

This is a problem because creating public buckets is a common use case for hosting static sites and assets - this workflow is so common that it is one of the featured workflows for S3.

Let's illustrate what the change looks like by trying to create a public bucket. If you're using the AWS console, you now must explicitly disable the "Block all public access" toggle as well as check a consent dialogue to create a bucket.

Note that this is for ensuring that a bucket can be public - you still need to apply the proper object or bucket-level policies to make objects public.

If you're a DevOps wizard that never touches the console, you might create a new bucket using Terraform or the CDK.

In this case, you might write the following code:

const publicBucket = new s3.Bucket(this, 'MyPublicBucket', {
  publicReadAccess: true,
});

When you run a diff, things look normal

$ cdk diff
Stack AutoCdkStack
IAM Statement Changes
┌───┬─────────────────────────┬────────┬──────────────┬───────────┬───────────┐
│   │ Resource                │ Effect │ Action       │ Principal │ Condition │
├───┼─────────────────────────┼────────┼──────────────┼───────────┼───────────┤
│ + │ ${MyPublicBucket.Arn}/* │ Allow  │ s3:GetObject │ AWS:*     │           │
└───┴─────────────────────────┴────────┴──────────────┴───────────┴───────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Resources
[+] AWS::S3::Bucket MyPublicBucket MyPublicBucketBCB7A648 
[+] AWS::S3::BucketPolicy MyPublicBucket/Policy MyPublicBucketPolicyDE00CFED 


✨  Done in 3.94s.

Try deploying it, however, and you run into an unhelpful "Access Denied" error with no further context

$ cdk deploy

✨  Synthesis time: 2.88s

AutoCdkStack:  start: Building 46900dc420169a4d5dc632927ff6ca3ba8c1ca93cdd60512a3c0950e2d75
...
AutoCdkStack: deploying... [1/1]
AutoCdkStack: creating CloudFormation changeset...
7:37:38 AM | CREATE_FAILED        | AWS::S3::BucketPolicy | MyPublicBucketPolicyDE00CFED
API: s3:PutBucketPolicy Access Denied

 ❌  AutoCdkStack failed: Error: The stack named AutoCdkStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: API: s3:PutBucketPolicy Access Denied
...
The stack named AutoCdkStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: API: s3:PutBucketPolicy Access Denied

The error comes from the change in default bucket policy - when you create a new bucket and want to make it public, you also must disable the "Block Public Access" settings on the bucket.

In our cdk example, this would look like the following

    new s3.Bucket(this, 'MyPublicBucket', {
      publicReadAccess: true,
      blockPublicAccess: { 
	      blockPublicAcls: true, blockPublicPolicy: false, 
	      ignorePublicAcls: true, restrictPublicBuckets: false},
    });

Unfortunately, this isn't well documented in the CDK docs. You will also run into similar problems when trying to provision a public bucket using Terraform. As of 2023-06-10, Both Terraform and CDK have S3 bucket creation tickets as the number one pinned issues in their respective repositories, here and here.

AWS CDK

Terraform

While the issue with public S3 Buckets seems minor, it is just the tip of the iceberg in terms of complexity. Creating a public/private bucket is a trivial example. Things become more complicated once you involve encrypted objects, other AWS services, and cross-account access. I wrote an article many moons ago about debugging eight different permission layers to diagnose an S3 bucket permission issue. Since that article was written, S3 has added even more complexity features like object lambdas and access points that add more layers to an already complicated access control model.

Zooming out - S3 is a single service. AWS has over 200 services. Each service is complicated by itself but that complexity compounds when they need to talk to one another. It then further compounds once you involve multiple accounts.

So what do we do about this?

Some solutions address the complexity by building platforms that abstract away the cloud provider. Services like Heroku and Vercel do a great job to improve DX outside of the cloud. They provide this DX at the cost of control and extensibility (plus significant markups in pricing).

Other solutions address the complexity by building better tooling inside the cloud. Infrastructure-as-Code (IaC) primitives like Terraform, CDK, and Pulumi make it easier to programmatically create infrastructure and replicable environments. Unfortunately, they still require users to deal with all the messy implementation details of each service. The latest addition in the IaC space is Winglang, a new project made by the original creator of the AWS CDK. It aims to be a higher-level IaC primitive that further abstracts many of the messy details of cloud services and is to terraform what Python is to C.

The change in S3 permission defaults highlights the challenges of building a cloud that is simple, secure, and extensible. This can feel like a CAP theorem-like problem but there are no laws of physics that prevent us from having all three. I think there's a lot more that can be done in this space, particularly the approach of improving the DX inside the cloud vs offering a limited experience outside of it. This is a space I have a few ideas on which I aim to share in the coming weeks. Meanwhile, may your clouds be unfettered and your buckets secure (even if they are not simple).

NOTE: This is Part II of a blog series that explores writing code with LLMs. See here for Part I of the series.

This post continues where we left off in Part I. It builds on the previous work to generate a fully functional vscode extension without human intervention.

All the code used in this experiment is open source and published on GitHub. Specifically:

• A fork of smol-ai customized for creating VS Code extensions: smol-vscode-developer

• VS Code extension: repository, published extension

Recap

Previously, we used GPT4 to generate code to create a vscode extension from scratch. This involved lots of iteration and manual intervention. Most of the issues came from a lack of coherence (variables generated in one file were named something different in another file) and GPT's knowledge cutoff date (it's only been trained on data that was available in September 2021).

To make up for these deficiencies, we start with an updated prompt that accounts for all the issues we encountered in part I.

Prompt from part I:

the app is: a vscode extension that lets the user adjust the heading level of the selected text. it should have three commands: increase heading level, decrease heading level, and set heading level

Important details:

- make sure there is a tsconfig.json file present
- created a shared `utils.ts` file that exports the `adjustHeadingLevel` method to adjust headings
- in `package.json`, use the `1.6.1` dependency for `vscode-test`
- make sure to include `.vscode/launch.json` as one of the mandatory files
    - it should have a "extensionHost" launch task
    - it should have "--disable-extensions" as one of the "args"

Updated Prompt:

a vscode extension that lets the user adjust the heading level of the selected text. it should have three commands: increase heading level, decrease heading level, and set heading level
created a shared `utils.ts` file that exports the `adjustHeadingLevel` method to adjust headings
when registering commands, use an identifier that would match the following regex: "extension.(increaseHeading|decreaseHeading|setHeading)"


Important details:
- make sure the following files are present:
    - tsconfig.json
    - .gitignore
    - README.md
    - src/test/runTest.ts
- make sure the following constraints are observed in ".gitignore"
    - ignore javascript files
- make sure the following constraints are observed in "runTest.ts"
    - import "mocha" directly (eg. `import Mocha from 'mocha'`)
    - import glob as named export (eg. `import {glob} from 'glob'`)
    - do not import modules that are not needed
    - use node builtin "assert" library for assertions
    - DO NOT use parameters that have implicit 'any' type - add proper types or add an explicit 'any' type
- make sure the following shared dependencies are present
    - mocha for testing
- make sure that the README has the following sections:
    - Overview: what this project is about
    - Quickstart: how to use this extension
    - Development Guide: how to develop this extension
- make sure the following dependencies are present in `package.json`
    - @vscode/test-electron: ^2.3.2
    - glob: ^7.1.4
    - dependencies required for testing (eg. mocha)
    - dependencies required for types (eg. @types/mocha, @types/glob)
- make sure the following dependencies are not present in `package.json`
    - "vscode-test"
- make sure the following constraints are observed in `package.json`
    - when referring to an extension command, use the format "extension.{command-name}". DO NOT use the extension name as the command prefix
    - when adding "activationEvents", do not include "onCommand" directives
- make sure to include `.vscode/launch.json` as one of the mandatory files
    - it should have a "extensionHost" launch task
    - it should have "--disable-extensions" as one of the "args"
    - the "preLaunchTask" task should be "npm: compile"
- make sure that you have tests for the user-specified functionality
- for all typescript code, generate correct imports according to `"esModuleInterop": true` is set in `tsconfig.json`
    - when exporting modules, ALWAYS use a named export, NEVER a default export. if the module is the only export in the file, the module name MUST BE IDENTICAL to the filename 
    - when importing modules that we have written, ALWAYS import the names of modules we are using
    - when importing modules, ONLY import modules that will be used 
    - when calling a function from 'shared_dependencies', ALWAYS use the given type signature

There are a lot of changes - let's go over each section and explain what's happening.

Specifying Files

We start by specifying all the files that GPT might miss. Files like '.gitignore' are not core to creating a vscode extension but necessary for developers to sanely manage a git repo. We include them directly in the prompt since we don't want developers to have to add these "boilerplate" details every time.

- make sure the following files are present:
    - tsconfig.json
    - .gitignore
    - README.md
    - test/runTest.ts

Specifying File Contents

We also need to add additional constraints depending on the file.

For example, the .gitignore won't always ignore javascript unless explicitly specified in the prompt.

- make sure the following constraints are observed in ".gitignore"
    - ignore javascript files

We also encounter issues when generating a README - whether or not it covers setup instructions is not deterministic. We can make it so by being prescriptive over what sections we expect.

- make sure that the README has the following sections:
    - Overview: what this project is about
    - Quickstart: how to use this extension
    - Development Guide: how to develop this extension

Specifying Dependencies

We explicitly list out dependencies that GPT might miss. These involve dependencies introduced after GPT's cutoff date (eg. @vscode/test-electron) and development dependencies

- make sure the following shared dependencies are present
    - mocha for testing
- make sure the following dependencies are present in `package.json`
    - @vscode/test-electron: ^2.3.2
    - glob: ^7.1.4
    - dependencies required for testing (eg. mocha)
    - dependencies required for types (eg. @types/mocha, @types/glob)

NOTE: Having examples matters. Without the (eg. ...) in parenthesis, GPT generates NO dev dependencies

    - @vscode/test-electron: ^2.3.2
-   - dependencies required for testing 
+   - dependencies required for testing (eg. mocha)
-   - dependencies required for types 
+   - dependencies required for types (eg. @types/mocha)

We also list dependencies that GPT should not include. These are dependencies that have been deprecated since GPT's cutoff date.

- make sure the following dependencies are not present in `package.json`
    - "vscode-test"

Specifying Custom Constraints

We explicitly tell GPT about changes that are not in its index. In this case, the fact that VSCode no longer requires onCommand to be part of the extension manifest. We also prescribe how extensions are identified in the app.

- make sure the following constraints are observed in `package.json`
    - when referring to an extension command, use the format "extension.{command-name}". DO NOT use the extension name as the command prefix
    - when adding "activationEvents", do not include "onCommand" directives

NOTE: Order matters. Using the mental model of "if this then that" helps with specifying constraints. For the diff below, the first directive did not work but the second does.

-   - do not include "onCommand" directives when adding "activationEvents"
+   - when adding "activationEvents", do not include "onCommand" directives

We also need to explicitly tell GPT about the preLaunchTask as it has the tendency to generate npm: watch when left to its own devices. Because the watch command never exits, launching the app in development mode would hang otherwise

- make sure to include `.vscode/launch.json` as one of the mandatory files
    - it should have a "extensionHost" launch task
    - it should have "--disable-extensions" as one of the "args"
    - the "preLaunchTask" task should be "npm: compile"

Specifying Code

We explicitly tell GPT about what coding style to use. This is because GPT has no context for the contents of the files that it generates. Javascript has multiple ways of exporting a module (eg. named export vs default export). GPT will more likely than not use one style for exporting and a different one for importing.

- for all typescript code, generate correct imports according to `"esModuleInterop": true` is set in `tsconfig.json`
    - when exporting modules, ALWAYS use a named export, NEVER a default export. if the module is the only export in the file, the module name should be identical to the filename
    - when importing modules that we have written, ALWAYS import the names of modules we are using
    - when importing modules, ONLY import modules that will be used 

Specifying Type Signatures

We provide additional details when specifying shared dependencies. Just generating the function names is not enough as GPT will hallucinate different ways of calling the shared function.

The following is a modification of the original prompt to generate explicit type annotations for any shared functions.

...
 Now that we have a list of files, we need to understand what dependencies they share.
 Name and briefly describe what is shared between the files we are generating, including exported variables, data schemas, and function signatures.
 Exclusively focus on the names of the shared dependencies, and do not add any other explanation. 
+ For function signatures, include the function name and the input and output parameters. Add type annotations to the function signatures

This adds the following annotation to shared functions.

Shared dependencies:
- - adjustHeadingLevel (function name in utils.ts)
+ - adjustHeadingLevel function signature: adjustHeadingLevel(text: string, adjustment: number): string

NOTE: If you are generating code that does not have type signatures, you can still tweak the prompt to generate "typescript types for shared dependencies"

Specifying Test Details

We need to specify many more implementation details for testing. There are more changes here than in other sections for the following reasons:

• VSCode made a major change in how tests should be written (after GPT's knowledge cutoff)

• the libraries used in testing (eg. glob and mocha) have changed major versions and function signatures since GPT knowledge cut off

• GPT does not generate dependencies for tests unless explicitly prompted

The following prompt tries to address most of these issues. We hardcode example values and get very prescriptive about what to do and not do

- make sure the following constraints are observed in "runTest.ts"
    - import "mocha" directly (eg. `import Mocha from 'mocha'`)
    - import glob as named export (eg. `import {glob} from 'glob'`)
    - do not import modules that are not needed
    - use node builtin "assert" library for assertions
    - DO NOT use parameters that have implicit 'any' type - add proper types or add an explicit 'any' type
- make sure that you have tests for the user-specified functionality

NOTE: Despite the hints, the tests unfortunately still do not compile most of the time. I timeboxed how long I could spend getting this to work which means getting GPT to correctly generate tests will be an exercise left to the reader

Results

Running the prompt at the top of this writeup should result in a working vscode extension (most of the time). You can pull down the code from here and follow the README to try it out yourself.

NOTE: The generated tests might need manual tweaking - the extension itself should function.

You can find the output of our generation here. This is also published in the vscode marketplace and you can download the extension here to try it out yourself.

To check if this would work for a different task, I also generated an extension to format markdown. The prompt is as follows:

a vscode extension that lets the user bold, italicise and strikethrough the selected markdown text. it should have three commands: "bold selection", "italicise selection", and "strikethrough selection"
when registering commands, use an identifier that would match the following regex: "extension.(boldSelection|italiciseSelection|strikethroughSelection)"

...

The ... contains the same prompt details that we covered in this write-up. This worked on the first try (with the caveat that the tests do not and require tweaking)

Code for this extension here

Lessons Learned

• little changes in the prompt have big changes in the outcome

  • ORDER Matters. GPT is a next-word predictor. It reads context from left to right. When you want GPT to do something, tell it when it should act before what it should do (eg. "when X happens, do Y" works much better than "do Y when X happens")

  • Examples help. Instead of saying include test dependencies, bias with an example include test dependencies (eg. mocha)

• watch out for non-deterministic outputs

  • Be prescriptive when there is ambiguity. Treat GPT as a smart intern. Tell it exactly what to do, especially when there are multiple ways of doing something (eg. how to import/export modules in javascript)

  • Use types. This builds on being prescriptive. When you want GPT to reuse a function, make sure the context includes the function signature

• do not trust and always verify

  • Verify the output. Ideally using automated tests. Even when you're explicit about everything, there is still a chance that GPT makes things up. In our case, even when the type signature of adjustHeadingLevel was set, GPT still generated an invalid usage of the function when creating a unit test

Parting Thoughts

The approach we took to generate code is what I call the "do-then-check" strategy. We tell GPT to do a thing, GPT does the thing, we check if the thing is right, and then we add anything that wasn't right as additional instructions to the prompt so that GPT does the thing right the next time.

Getting a functional solution for even a basic app using this strategy requires a huge amount of context. This is to overcome the following problems:

• dealing with information that is not in the LLM index (past its knowledge cutoff date)

• dealing with "coherence" - making sure that code generated across different runs did not conflict

Both issues reduce to the same issue - how do you find the right information to add to a limited context window?

Coincidentally, this is a problem space that I've been working in for the past decade. My initial startup was built around the mission of helping humans organize and make sense of information. Given that the information we have available far exceeds our own context window, how do we find the right information when needed?

I strongly believe that these problems are related. It is likely that the same means by which humans can deal with information overload also apply to LLMs.

Much more on this topic in the near future 😇

Lately, I've been playing around with LLMs to write code. I find that they're great at generating small self-contained snippets. Unfortunately, anything more than that requires a human to evaluate LLM output and come up with suitable follow-up prompts. Most examples of "GPT wrote X" are this - a human serves as a REPL for the LLM, carefully coaxing it to a functional result. This is not to undersell this process - it's remarkable that it works. But can we go further? Can we use an LLM to generate ALL the code for a complex program ALL at once without any human intervention?

Writing a VSCode Extension

To test GPT-4's ability to generate a complex program, I prompted it to create a VSCode extension that lets the user adjust the heading level of selected Markdown text. This task requires:

- Domain-specific knowledge about how to scaffold and expose a program to VSCode

- Mixing multiple languages and platforms: VS Code extensions are written in TypeScript, which requires writing configuration for Typescript, Node.js, and VSCode

- Generating multiple files

- Generating scaffolding to debug, build, and run code

Setup

For this experiment, I used GPT-4 for all generation purposes. I find it to be the most effective among current-day models.

In addition, I make use of the smol-ai framework to generate code.

smol-ai description from the README:

This is a prototype of a "junior developer" agent (aka smol dev) that scaffolds an entire codebase out for you once you give it a product spec, but does not end the world or overpromise AGI. instead of making and maintaining specific, rigid, one-shot starters, like create-react-app, or create-nextjs-app, this is basically create-anything-app where you develop your scaffolding prompt in a tight loop with your smol dev.

I like smol-ai because of its simplicity. The entire code generation logic is in a single Python file consisting of three primary functions:

1. generate a list of files that are needed to carry out the prompt (eg. package.json, index.js, ...)

2. generate a list of shared dependencies that are needed to carry out the prompt (eg. axios, react, ...)

3. for each file in the generated file list, generate code that would go into the file, making use of shared dependencies if applicable

Note that smol-ai also provides other capabilities like the ability to debug errors and generate a prompt based on the current code base. These are not explored in this post.

Prompt

I started with a basic prompt describing my problem statement

python main_no_modal.py "a vscode extension that lets the user adjust the heading level of the selected text. it should have three commands: increase heading level, decrease heading level, and set heading level"

Note that the prompt is short and general. I wanted to see how well GPT could do without feeding in any additional details or domain-specific knowledge beyond what was in the training set.

Output

After a few minutes spent generating code, smol-ai created the following files:

.
├── extension
│   ├── package.json
│   ├── src
│   │   ├── commands
│   │   │   ├── decreaseHeadingLevel.ts
│   │   │   ├── increaseHeadingLevel.ts
│   │   │   └── setHeadingLevel.ts
│   │   └── extension.ts
└── shared_dependencies.md

Note that shared_dependencies.md is created as a best guess of what common dependencies will be. In our case, it created a file with the following contents:

the app is: a vscode extension that lets the user adjust the heading level of the selected text. it should have three commands: increase heading level, decrease heading level, and set heading level

the files we have decided to generate are: tsconfig.json, package.json, utils.ts, extension.ts

Shared dependencies:

1. Exported variables:
   - adjustHeadingLevel (from utils.ts)

2. Data schemas:
   - None

3. ID names of DOM elements:
   - None

4. Message names:
   - None

5. Function names:
   - increaseHeadingLevel
   - decreaseHeadingLevel
   - setHeadingLevel

6. Dependencies:
    - None

Code

extension.ts is the entry point of a vscode extension. In it, we register the three commands that we specified in the prompt.

import * as vscode from 'vscode';
import { increaseHeadingLevel } from './commands/increaseHeadingLevel';
import { decreaseHeadingLevel } from './commands/decreaseHeadingLevel';
import { setHeadingLevel } from './commands/setHeadingLevel';

export function activate(context: vscode.ExtensionContext) {
    context.subscriptions.push(
        vscode.commands.registerCommand('extension.increaseHeadingLevel', increaseHeadingLevel),
        vscode.commands.registerCommand('extension.decreaseHeadingLevel', decreaseHeadingLevel),
        vscode.commands.registerCommand('extension.setHeadingLevel', setHeadingLevel)
    );
}

export function deactivate() {}

Below is the logic inside of decreaseHeadingLevel.ts

import * as vscode from 'vscode';

export async function decreaseHeadingLevel() {
    const editor = vscode.window.activeTextEditor;
    if (!editor) {
        return;
    }

    const document = editor.document;
    const selection = editor.selection;
    const selectedText = document.getText(selection);

    const headingRegex = /^(#+)\s(.*)$/;
    const match = selectedText.match(headingRegex);

    if (!match) {
        vscode.window.showErrorMessage('Invalid selection. Please select a heading to decrease its level.');
        return;
    }

    const currentLevel = match[1].length;
    if (currentLevel === 1) {
        vscode.window.showErrorMessage('Cannot decrease heading level. Already at the lowest level.');
        return;
    }

    const newLevel = currentLevel - 1;
    const newHeading = '#'.repeat(newLevel) + ' ' + match[2];

    await editor.edit((editBuilder) => {
        editBuilder.replace(selection, newHeading);
    });
}

The code checks for an active editor and if one exists, gets the selected text and looks for a markdown heading via regex. If a header is found and the current header level isn't already at the lowest level, it decreases the heading level.

At first glance, there is nothing amiss with the logic. It executes the command and checks for edge cases. It even provides helpful error messages which already puts it ahead of most human-generated programs...

Testing the extension

To test this extension, we need to successfully execute the following steps:

1. Install Dependencies

2. Compile Code

3. Run Extension

Step 1 - Install

We run into our first issue when trying to install dependencies.

$ yarn

Couldn't find any versions for "vscode-test" that matches "^1.6.2"
? Please choose a version of "vscode-test" from this list: (Use arrow keys)
❯ 1.6.1

Issue 1 - Couldn't find vscode-test

An inspection of package.json returns the following

{
  "name": "adjust-heading-level",
  ...
  "engines": {
    "vscode": "^1.62.0"
  },
  "devDependencies": {
    "@types/node": "^14.17.0",
    "@types/vscode": "^1.62.0",
    "typescript": "^4.4.2",
    "vscode": "^1.1.37",
    "vscode-test": "^1.6.2"
  },
}

The vscode engine determines the minimum version of vscode. The present-day (as of 2023-05-23) engine version is 1.78. The 1.62.0 version was released on October 21st, 2021

This corresponds to GPT4's knowledge cutoff date:

GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its data cut off (September 2021)

The vscode-test version of 1.6.2 seems suspiciously similar to 1.62 which indicates that GPT likely hallucinated the numbers.

In any case, this is easy enough to fix by specifying the correct version number and re-installing

-   "vscode-test": "^1.6.2"
+   "vscode-test": "^1.6.1"

Re-running the install process is successful the second time around.

$ yarn

...
[3/5] 🚚  Fetching packages...
[4/5] 🔗  Linking dependencies...
[5/5] 🔨  Building fresh packages...
✨  Done in 4.31s.

Step 2 - Build

Because typescript is a compiled language, we need to execute a build step to compile the code to javascript. The package.json comes with the following scripts:

"scripts": {
    "vscode:prepublish": "npm run compile",
    "compile": "tsc -p ./",
    "watch": "tsc -watch -p ./",
    "postinstall": "node ./node_modules/vscode/bin/install",
    "test": "npm run compile && node ./node_modules/vscode/bin/test"
  },

We can build the code by running the compile script. This is where we run into our next issue:

$ yarn compile
warning package.json: No license field
warning adjust-heading-level@0.1.0: The engine "vscode" appears to be invalid.
$ tsc -p ./
error TS5057: Cannot find a tsconfig.json file at the specified directory: './'.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Issue 2 - Cannot find a tsconfig.json

Typescript requires a tsconfig.json file to compile into javascript. If you remember our initial file layout, there is no tsconfig.json file.

.
├── extension
│   ├── package.json
│   ├── src
│   │   ├── commands
│   │   │   ├── decreaseHeadingLevel.ts
│   │   │   ├── increaseHeadingLevel.ts
│   │   │   └── setHeadingLevel.ts
│   │   └── extension.ts
└── shared_dependencies.md

We can remediate this by adding the config and re-building. But now we run into more issues:

$ tsc --init
$ yarn compile

src/commands/decreaseHeadingLevel.ts:1:25 - error TS2307: Cannot find module 'vscode' or its corresponding type declarations.

1 import * as vscode from 'vscode';
                          ~~~~~~~~

src/commands/decreaseHeadingLevel.ts:30:24 - error TS7006: Parameter 'editBuilder' implicitly has an 'any' type.

30     await editor.edit((editBuilder) => {
                          ~~~~~~~~~~~

src/commands/increaseHeadingLevel.ts:1:25 - error TS2307: Cannot find module 'vscode' or its corresponding type declarations.

...

Found 7 errors

Issue 3 - Cannot find modules

The reason typescript can't find the module vscode is because of the syntax we use for import statements:

// this is failing
import * as vscode from 'vscode';

// this would work
import vscode from 'vscode';

The reason for the different syntax comes from the differences between CommonJs and ES Modules and how they export dependencies as well as how typescript transpiles those exports. The maddening hellscape quirks in module compatibility can be a blog post unto itself - for now, we can fix the issue by disabling esModuleInterop inside of tsconfig.json

@@ -71,7 +71,7 @@
-    "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */
+    "esModuleInterop": false,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables 'allowSyntheticDefaultImports' for type compatibility. */

Note that esModuleInterop was changed to true by default since typescript 4.4. This was released on March 15, 2022 - after GPT4's knowledge cutoff date.

Let's try building again. Just one more error this time:

$ yarn compile

src/commands/setHeadingLevel.ts:2:10 - error TS2305: Module '"../extension"' has no exported member 'adjustHeadingLevel'.

2 import { adjustHeadingLevel } from '../extension';

Found 1 error.

Issue 4 - No exported member

This last error comes from trying to import a function that does not exist.

Specifically, the following logic in setHeadingLevel.ts:

import * as vscode from 'vscode';
import { adjustHeadingLevel } from '../extension';

export async function setHeadingLevel() {
  ...
}

GPT is prone to lie be optimistic about declaring its dependencies. It sometimes calls or imports functions that do not exist. This is one of those cases.

We can fix this by removing the dependency and manually adding the logic inside of setHeadingLevel

@@ -1,5 +1,4 @@
 import * as vscode from 'vscode';
-import { adjustHeadingLevel } from '../extension';
 
 export async function setHeadingLevel() {
     const editor = vscode.window.activeTextEditor;
@@ -14,6 +13,12 @@ export async function setHeadingLevel() {
         vscode.window.showErrorMessage('invalidSelection');
         return;
     }
+    const headingRegex = /^(#+)\s(.*)$/;
+    const match = selectedText.match(headingRegex);
+    if (!match) {
+        vscode.window.showErrorMessage('Invalid selection.');
+        return;
+    }
 
     const inputOptions: vscode.InputBoxOptions = {
         prompt: 'setHeadingLevelPrompt',
@@ -31,6 +36,16 @@ export async function setHeadingLevel() {
 
     if (headingLevel) {
         const newHeadingLevel = parseInt(headingLevel);
-        adjustHeadingLevel(editor, selection, selectedText, newHeadingLevel);
+    
+        const newHeading = '#'.repeat(newHeadingLevel) + ' ' + match[2];
+    
+        await editor.edit((editBuilder) => {
+            editBuilder.replace(selection, newHeading);
+        });
     }
 }

Note that most of the code was lifted from decreaseHeadingLevel.ts.

Lets build again. This time, it's successful 🎉

$ tsc -p ./
✨  Done in 0.80s.

But does it run?

Step 3 - Run

Note that GPT provided no instructions on how to run the extension. Or on how to install or build the extension for that matter. It is simple enough to do if you've built vscode extensions before but this can be a barrier to entry for newcomers.

Running a vscode extension requires that you go to the "Run and Debug" panel and launch the vscode extension task when the extension.ts file is active in the editor.

This launches a new vscode window with our extension installed. This also errors out as soon as I tried invoking a command.

Command 'Increase Heading Level' resulted in an error command 'adjust-heading-level. 'increaseHeadingLevel' was not found

Issue 5 - Commands not found

VSCode knows about commands when they are declared inside of package.json.

Our package.json declares the following commands:

  "activationEvents": [
    "onCommand:adjust-heading-level.increaseHeadingLevel",
    "onCommand:adjust-heading-level.decreaseHeadingLevel",
    "onCommand:adjust-heading-level.setHeadingLevel"
  ],
  ...
  "contributes": {
    "commands": [
      {
        "command": "adjust-heading-level.increaseHeadingLevel",
        "title": "Increase Heading Level"
      },
      {
        "command": "adjust-heading-level.decreaseHeadingLevel",
        "title": "Decrease Heading Level"
      },
      {
        "command": "adjust-heading-level.setHeadingLevel",
        "title": "Set Heading Level"
      }
    ]
  }

After declaring inside of package.json, these commands also need to be registered inside the extension.

Our extension.ts

export function activate(context: vscode.ExtensionContext) {
    context.subscriptions.push(
        vscode.commands.registerCommand('extension.increaseHeadingLevel', increaseHeadingLevel),
        vscode.commands.registerCommand('extension.decreaseHeadingLevel', decreaseHeadingLevel),
        vscode.commands.registerCommand('extension.setHeadingLevel', setHeadingLevel)
    );
}

Do you see the issue?

The typescript file declares commands as extension.{COMMAND} but the package.json declares them as adjust-heading-level.{COMMAND}

We can fix this by adjusting package.json to match the code. While the fix itself is simple, being able to diagnose the issue correctly takes some domain knowledge in knowing where to look.

@@ -1,5 +1,5 @@
 {
   "displayName": "Adjust Heading Level",
   "description": "A VSCode extension that lets the user adjust the heading level of the selected text.",
   "version": "0.1.0",
@@ -10,23 +10,20 @@
     "Other"
   ],
   "activationEvents": [
-    "onCommand:adjust-heading-level.increaseHeadingLevel",
-    "onCommand:adjust-heading-level.decreaseHeadingLevel",
-    "onCommand:adjust-heading-level.setHeadingLevel"
   ],
   "main": "./src/extension.js",
   "contributes": {
     "commands": [
       {
-        "command": "adjust-heading-level.increaseHeadingLevel",
+        "command": "extension.increaseHeadingLevel",
         "title": "Increase Heading Level"
       },
       {
-        "command": "adjust-heading-level.decreaseHeadingLevel",
+        "command": "extension.decreaseHeadingLevel",
         "title": "Decrease Heading Level"
       },
       {
-        "command": "adjust-heading-level.setHeadingLevel",
+        "command": "extension.setHeadingLevel",
         "title": "Set Heading Level"
       }
     ]

NOTE: I also used this to remove the activationEvents - these determine when a vscode extension trigger activates. For command-based activations, vscode is now able to detect them automatically and so no longer needs to be manually declared

Let's try running again and increasing the header level.

Well, that's not supposed to happen 😟

Issue 6 - The Decreasing Increase

Instead of increasing the header, we are getting the header level decreased.

Let's take a look at increaseHeadingLevel.ts

import * as vscode from 'vscode';

export async function increaseHeadingLevel() {
    const editor = vscode.window.activeTextEditor;
    if (!editor) {
        return;
    }

    const document = editor.document;
    const selection = editor.selection;
    const selectedText = document.getText(selection);

    const headingRegex = /^(#+)\s(.*)$/;
    const match = selectedText.match(headingRegex);

    if (!match) {
        vscode.window.showErrorMessage('Invalid selection. Please select a valid heading.');
        return;
    }

    const currentLevel = match[1].length;
    const newLevel = Math.max(1, currentLevel - 1);
    const newText = '#'.repeat(newLevel) + ' ' + match[2];

    await editor.edit((editBuilder) => {
        editBuilder.replace(selection, newText);
    });
}

Do you see the issue?

There is a bug caused by a single character diff.

@@ -19,7 +19,7 @@ export async function increaseHeadingLevel() {
     }
 
     const currentLevel = match[1].length;
-    const newLevel = Math.max(1, currentLevel - 1);
+    const newLevel = Math.max(1, currentLevel + 1);
     const newText = '#'.repeat(newLevel) + ' ' + match[2];

Let's compile and run it again.

It works 🥳

Thoughts

So how did we do? We got a working extension. We got it to accomplish the goal set out in our prompt.

The journey to this point was not "automatic". We ran into many issues along the way. Lacking prior knowledge of typescript, node.js and vscode, these issues would have taken a while to debug.

And even though the code we were able to generate working code, there are still many improvements to be made:

• there are no instructions on how to develop, use, or publish the extension

• there is no .gitignore for typescript/javascript/vscode artifacts

• there is no launch.json file that configures running the extension in development

• there are no tests

• there is no code reuse

Some Stats

GPT generated 9 files that cover ~100 lines of typescript, ~180 lines json, and 17 lines of markdown.

$ cloc --exclude-dir=node_modules,out --not-match-f=package-lock.json --not-match-f=prompt.md --include-ext=ts,json,md .
      15 text files.
      13 unique files.
       7 files ignored.

github.com/AlDanial/cloc v 1.92  T=0.01 s (986.5 files/s, 36610.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                             4              8              0            181
TypeScript                       4             22              0             98
Markdown                         1              8              0             17
-------------------------------------------------------------------------------
SUM:                             9             38              0            296
-------------------------------------------------------------------------------

The final file tree

$ tree --gitfile extension/.gitignore
.
├── extension
│   ├── package.json
│   ├── src
│   │   ├── commands
│   │   │   ├── decreaseHeadingLevel.ts
│   │   │   ├── increaseHeadingLevel.ts
│   │   │   └── setHeadingLevel.ts
│   │   └── extension.ts 
│   ├── tsconfig.json
│   └── yarn.lock
├── prompt.md
└── shared_dependencies.md

Out of the ~300 lines generated, we had to modify/add ~18 lines in order to make everything work.

Takeaways

GPT was able to generate most of the code using a naive prompt with no domain-specific context.

Some things to note:

• GPT4 does great with code in its index but will likely generate bad logic if the underlying specs have changed since its knowledge cutoff date (September 2021)

• GPT4 can hallucinate subtle bugs. In the increaseHeadingLevel.ts case, it was a one-character difference that caused the extension to do the exact opposite of what the command was supposed to do

• GPT4 is great at scaffolding boilerplate but domain expertise still matters (for now). This is especially true when building on the tech that has changed since GPT4's cutoff date

• GPT4 introduces yet another abstraction layer for programming. We now have 7 translation layers for the case of writing typescript (which can easily be doubled when involving containers or VMs) 🤦‍♂

  

Future Directions

I did the initial experiment with a naive general prompt and no additional context. There is lots of room for improvement. Some next steps:

• every issue that we encountered when trying to run the extension, include it as a detail in the prompt for GPT to watch out for

  • generalize this by indexing and summarizing the vscode extension docs to mitigate for new information that is not in GPT's current index

    • explore doing this by chaining an LLM that has access to today's context (eg. Bard) with GPT

• generate tests to validate logic and have GPT autocorrect itself if tests fail

• generate a checklist for what a high-quality vscode extension looks like and have GPT verify and autocorrect the artifacts it generates

NOTE: I've already run a subset of these steps and was able to get the error count to zero on the first generation. Will need to see if it generalizes to other examples. Look out for details in a future post

A Note to the reader

I'm a solo YC founder that is currently pivoting into building useful tools in the LLM space. If this is something that you're passionate about and want to be a part of, either as a cofounder (I'm looking for someone with a technical background in sales/product/go-to-market) or as a founding team member, please reach out.

Like everyone on the internet, I've been playing around with large language models (LLMs). As a result, it has changed the way I think and write code.

Today, I'm sharing how I start most of my coding sessions, by using GPT as a compiler to translate pseudocode into real code.

All examples below are generated using chatgpt-3.5

NOTE: while you generally get better results running chatgpt-4 instead of 3.5, I find the huge latency in responses to not be worth the incremental improvements that chatgpt-4 provides

What is Pseudocode

Pseudocode is a plain language description of the steps in an algorithm or another system. Pseudocode often uses structural conventions of a normal programming language but is intended for human reading rather than machine reading. Source: Wikipedia

Pseudocode uses plan language to describe a particular algorithm to humans. The output is prose that is easily understood by humans but not executable by machines.

Large language models are trained to take plain language intended for humans and generate effective outputs from the description. Code, aka instructions that can be executed by machines, is one of those outputs.

So what if we combine them? Let's have our cake and eat it too!

Why Pseudocode

Normally, when you see examples of GPT-generated code, they come in the form of simple directives like "fetch the top 50 items from hacker news" or "get the stock price for Google".

While these directives do generate executable code (most of the time), they do not enforce fine-grained constraints. Constraints arise in most real-world applications - maybe you're working with an existing code base that limits what dependencies you can use or you have to accomplish a complicated workflow in a particular sequence.

Pseudocode works well when in these situations and allows you, the programmer, to have all the benefits of LLMs while still retaining control of the output.

Case Study: Fetching All the AWS Docs

Today's example goes over the actual workflow I used to create AWS Reference Notes - this project attempts to consolidate all known gotchas, limitations and caveats of various AWS services.

AWS docs are available in public git repositories. As part of the project, I needed to implement logic that could clone these repos into a specific directory layout. I also needed the code to use the isomorphic-git library which allows git to be run both on the web as well as on a server.

Prompt 1

My initial prompt to gpt.

Convert the given pseudocode into valid typescript. 

The following third-party libraries will be available:
- isomorphic-git as git

The following operators are natural language operations. Translate these directives into valid typescript. Make sure to implement all logic. 
- `%{action}%`: implement the desired {action} using what you know about the language and available libraries
- `%{library}|{action}%`: implement the desired {action} using the suggested {library}. you MUST use functions available in {library}. if none exist, create a helper function that accomplishes the desired {action}

Pseudocode below:
```ts
export main(services: string[]) {
    %set basedir to `__dirname/..`%
    basedir = 

    await Promise.all services.map( async service => {
        await upsertDevGuide({service, basedir})
    })
}

awsgitrepo(service: string) {
    return `https://github.com/awsdocs/${service}-developer-guide.git`
}

upsertDevGuide(opts: {service: messtring, basedir: string}) {
    guidePath = path.join(basedir, "docs", service, "developer-guide")
    if !(
        %git|check if "guidePath" exists and is a git repo%

    ) {
        %ensure that the directory for "guidePath" exists, create if it does not%
        %git|run git clone at "guidePath" using output of "awsgitrepo(service)"%

    } else {
        %git|run git pull at "guidePath"%
    }
}
```

This prompt has 3 directives:

1. specify the desired output format (typescript).

2. specify the availability of third-party libraries (isomorphic-git)

3. specify natural language operations (NLOs) that we want GPT to translate into code.

There are two NLOs in this example:

• %{action}%: perform any action (wildcard action NLO)

• %{library}|{action}%: perform an action using the suggested library (library action NLO)

The wildcard nlo is useful when you want a specific side effect or state to happen that requires multiple steps (eg. ensuring a directory exists). The library nlo is useful when you know a library has a particular function and you can't be bothered to fill in all the boilerplate.

Response 1

Here's the conversion of the given pseudocode into valid TypeScript, taking into account the natural language operations:

```typescript
import * as git from 'isomorphic-git';
import * as path from 'path';

export async function main(services: string[]) {
  const basedir = path.join(__dirname, '..');

  await Promise.all(services.map(async (service) => {
    await upsertDevGuide({ service, basedir });
  }));
}

function awsgitrepo(service: string) {
  return `https://github.com/awsdocs/${service}-developer-guide.git`;
}

async function upsertDevGuide(opts: { service: string, basedir: string }) {
  const guidePath = path.join(opts.basedir, 'docs', opts.service, 'developer-guide');
  
  const isGitRepo = await git.isGitRepo({ fs, dir: guidePath }).catch(() => false);

  if (!isGitRepo) {
    await ensureDirectoryExists(guidePath);
    await git.clone({ dir: guidePath, url: awsgitrepo(opts.service) });
  } else {
    await git.pull({ fs, dir: guidePath });
  }
}

async function ensureDirectoryExists(directoryPath: string) {
  const fs = git.plugins.get('fs');
  const dirPath = path.dirname(directoryPath);
  
  // Create directory if it doesn't exist
  await fs.promises.mkdir(dirPath, { recursive: true });
}
```

Please note that in order to use `isomorphic-git` and its functions, you need to have it installed as a dependency in your TypeScript project.

The initial response from gpt doesn't compile but gets us most of the way there. It has the following issues:

• git operations require additional arguments (http and fs parameters)

• git.isGitRepo is a hallucination and not a real function

• ensureDirectoryExists uses git.plugins.get('fs') (which might be hallucinated) instead of the builtin fs module

Prompt 2

When first encountering issues, it's worthwhile to ask GPT to make corrections.

isGitRepo isn't a function

Note that the follow-up is much more concise than the initial prompt. GPT already has the prior context so only minimal prompting is needed to "fine-tune" the results.

Response 2

Apologies for the confusion. You're correct that `isGitRepo` is not a built-in function in `isomorphic-git`. Instead, we can check if the `.git` directory exists in the specified path to determine if it's a git repository. Here's the updated code:

```typescript
import * as git from 'isomorphic-git';
import * as fs from 'fs';
import * as path from 'path';

export async function main(services: string[]) {
  const basedir = path.join(__dirname, '..');

  await Promise.all(services.map(async (service) => {
    await upsertDevGuide({ service, basedir });
  }));
}

function awsgitrepo(service: string) {
  return `https://github.com/awsdocs/${service}-developer-guide.git`;
}

async function upsertDevGuide(opts: { service: string, basedir: string }) {
  const guidePath = path.join(opts.basedir, 'docs', opts.service, 'developer-guide');
  
  const isGitRepo = await checkIfGitRepo(guidePath);

  if (!isGitRepo) {
    await ensureDirectoryExists(guidePath);
    await git.clone({ dir: guidePath, url: awsgitrepo(opts.service) });
  } else {
    await git.pull({ dir: guidePath });
  }
}

async function checkIfGitRepo(directoryPath: string): Promise {
  const gitDirPath = path.join(directoryPath, '.git');

  return new Promise((resolve) => {
    fs.access(gitDirPath, fs.constants.F_OK, (err) => {
      resolve(!err);
    });
  });
}

async function ensureDirectoryExists(directoryPath: string) {
  const dirPath = path.dirname(directoryPath);

  // Create directory if it doesn't exist
  await fs.promises.mkdir(dirPath, { recursive: true });
}
```

This updated code checks if the `.git` directory exists in the specified path to determine if it's a git repository. If the `.git` directory exists, it is considered a git repository; otherwise, it is not.

Since GPT generates the entire code with the corrections, it can be hard to spot the changes. I've highlighted them in the diff below.

diff --git a/src/pseudocode.gpt35.1.ts b/src/pseudocode.gpt35.1.ts
index 6977cae..425cc2a 100644
--- a/src/pseudocode.gpt35.1.ts
+++ b/src/pseudocode.gpt35.1.ts
@@ -1,4 +1,5 @@
 import * as git from 'isomorphic-git';
+import * as fs from 'fs';
 import * as path from 'path';
 
 export async function main(services: string[]) {
@@ -16,20 +17,29 @@ function awsgitrepo(service: string) {
 async function upsertDevGuide(opts: { service: string, basedir: string }) {
   const guidePath = path.join(opts.basedir, 'docs', opts.service, 'developer-guide');
   
-  const isGitRepo = await git.isGitRepo({ fs, dir: guidePath }).catch(() => false);
+  const isGitRepo = await checkIfGitRepo(guidePath);
 
   if (!isGitRepo) {
     await ensureDirectoryExists(guidePath);
     await git.clone({ dir: guidePath, url: awsgitrepo(opts.service) });
   } else {
-    await git.pull({ fs, dir: guidePath });
+    await git.pull({ dir: guidePath });
   }
 }
 
+async function checkIfGitRepo(directoryPath: string): Promise<boolean> {
+  const gitDirPath = path.join(directoryPath, '.git');
+
+  return new Promise<boolean>((resolve) => {
+    fs.access(gitDirPath, fs.constants.F_OK, (err) => {
+      resolve(!err);
+    });
+  });
+}
+
 async function ensureDirectoryExists(directoryPath: string) {
-  const fs = git.plugins.get('fs');
   const dirPath = path.dirname(directoryPath);
-  
+
   // Create directory if it doesn't exist
   await fs.promises.mkdir(dirPath, { recursive: true });
 }

Note that GPT was able to fix the issue with the hallucinated git.isGitRepo function. In the process, it also simplifies the ensureDirectoryExists and switched to the node fs dependency. That said, it also went one step back by removing the (required) fs parameter from the git.pull function.

Finishing it up.

With more prompting, we can get GPT to fix the remaining issues with the code. I find that this isn't worthwhile as it becomes simpler to manually apply the remaining fixes at this point.

index 425cc2a..3bff3d9 100644
--- a/src/pseudocode.gpt35.1.ts
+++ b/src/pseudocode.gpt35.1.ts
@@ -1,6 +1,7 @@
 import * as git from 'isomorphic-git';
 import * as fs from 'fs';
 import * as path from 'path';
+import http from 'isomorphic-git/http/node';
 
 export async function main(services: string[]) {
   const basedir = path.join(__dirname, '..');
@@ -21,9 +22,9 @@ async function upsertDevGuide(opts: { service: string, basedir: string }) {
 
   if (!isGitRepo) {
     await ensureDirectoryExists(guidePath);
-    await git.clone({ dir: guidePath, url: awsgitrepo(opts.service) });
+    await git.clone({ dir: guidePath, url: awsgitrepo(opts.service), fs, http });
   } else {
-    await git.pull({ dir: guidePath });
+    await git.pull({ fs, dir: guidePath, http });
   }
 }

As you can see, only 3 lines needed to be changed to arrive at the working code.

Next Steps

There's a lot more to talk about that didn't make it in this article. Some things I'm currently experimenting with:

• enforcing high-level styles (eg. prefer object-based arguments vs positional arguments)

• additional operators (eg. A := B: extract anywhere from object operator. This extracts property "A" from "B". This requires inferring a path that will yield the value "A" from "B")

• automatically loading in additional context from the existing codebase

There's no end to things that you can do here - context size's the limit. If you have interesting examples of using GPT for code, please share them either in the comments below or reach out at kevin(a t)dendron.so. I'm currently thinking about putting together a repo to collect examples of how people code using GPT and looking for folks that are interested in contributing 🌱

An answer is a dead thing. A snapshot at a point in time.

Yes. No. Maybe.

I agree. I disagree.

This is good. This is bad.

Answers close of possibilities. The collapse infinity into a singularity. When people hold on to answers, it becomes dogma.

What are questions?

A question is a moving target. An invitation to be curious.

What?

Why?

How?

When?

Questions represent possibilities. They ask you to take in what is and imagine what could be. When people consider a question, they embark on a journey.

So ask yourself.

Who are you?

What do you stand for?

What do you yearn to do?

When will you start?

Dear Founder,

Burnout sucks. It is an insidious parasite that festers underneath the surface. You might not even notice it growing until the day you wake up and realize that you are underwater.

Burnout robs you of your joy, your will, and your abilities to do your highest calling.

And even if you are aware of both the fact and the consequence - more likely than not, you will continue and push forward. Because to do otherwise would mean acknowledging that you are burned out. And that weight is too much to bear. So you put off the day of reckoning and continue working. But the burnout is still there. Collecting interest. And it will come due.

The Two Phases of a (Venture-Backed) Startup

Burnout often happens in the "early" part of the startup journey.

This part is marked by two distinct phases:

• the phase before product market fit (PMF)

• the phase after PMF

The action of pushing a boulder up a hill is used to describe the phase before PMF. It is grueling work and you're not quite sure who will win in the end - you or the boulder.

The action of chasing a boulder down a hill is used to describe the phase after PMF. This is a different but equally hard sort of work as you run as fast as you can just to keep up with boulder.

This letter deals with burnout that can occur pushing the boulder up the hill.

The Burnout

I came across a recent hacker news comment that described burnout as an inequality between "resources", "agency", and "care".

Resources describe what you have at your disposal to achieve a particular outcome - this usually is some combination of personal capabilities, capital, and team.

Agency describes how much control you have over the outcome.

Care describes how invested you are in the outcome.

In big tech, the regular worker has access to near-unlimited resources but often very little agency. In a startup, the opposite is true.

People can burn out in big tech because they care about the outcome but have little agency in controlling it. In startups, founders burn out because they care about the outcome but run out of the resources necessary to achieve it.

As a startup founder, you make up for the lack of resources by being agile. By doubling down in a few key areas and executing extremely well. By putting in sweat, blood, and tears to push your idea forward.

In the phase before PMF, all your effort is concentrated on pushing the boulder up the hill.

When Push Leads to Burnout

Burnout happens when you push the boulder up the hill for too long with little progress and no end in sight.

Maybe you're pushing the wrong boulder (your boulder is made out of lead). Maybe you're on the wrong hill (consider going up the one that isn't covered in snow). Whatever the reason - you come to realize that this boulder is not going anywhere.

Meanwhile, you've already spent years (and capital) working on this boulder. Your "resources" are running low and prospects are drying up. What do you do?

This depends on your mental model of "doing a startup".

Striking Gold

One mental model of doing startups is that of "looking for gold".

Imagine a prospector searching for gold. He or she will continue forward because success comes with finding the gold. There's always the chance that the next dig will lead to the mother lode and they can exhaust all options to get to that outcome.

In startup lingo, replace the dig with the next feature launch. The next enterprise customer. The next influencer campaign.

In the "looking for gold" model, you are working for a singular outcome that will make you whole. Once you achieve it, all your problems go away.

This isn't the case for startups. Say you do "find gold" and make it to the top of the hill. Your journey is not done. Finding PMF means that it is just beginning. Now the hard work of chasing that boulder down the hill begins.

The average SaaS company takes 9 years to exit. Even if you think you can push that boulder to the top by sheer force of will, can you continue for another five to ten years if you've already burned yourself out getting there?

Infinite Games

Another mental model of doing startups is that of an "infinite game".

The concept of an infinite game is discussed in the book Finite and Infinite Games. A finite game has an end and players play to end the game. An infinite game has no end, players play to continue the game.

While venture-backed startups can seem like a finite game with a set end (aka exit) in mind, it's not helpful to think of them as such.

This is because most startups fail - if you are trying to measure success by the outcome of any one startup, it is like a gambler going all in on a single round of roulette.

If you do your startup with this mindset, you will continue pushing that boulder even if it is made out of lead. Regardless of whether you make it past the hill, you will probably burn through all your resources getting there, causing the "game" to end.

By "game", I'm not talking about the current manifestation of your startup. Rather, I'm talking about your spirit. Your passion. Your ability to continue playing the game.

Your odds of "winning" the game become much higher if you play the "infinite game". In doing so, you detach yourself from the outcome and invest in the journey. It means being willing to fail, learn from mistakes, and start over if need be. It means taking care of your "resources" because you are playing the "infinite game".

Taking a break

If you identify yourself as someone that has pushed a lead boulder up a snowcapped mountain for much too long, you have a few options.

You can keep pushing and hope you strike gold soon. And hope that in your current state, you can last another five to ten years chasing that boulder.

Or you can choose to step away. Take a break. This means walking away from your startup for some time. A week. A month. Maybe even a year. Whatever you have to do to restock your resources and rekindle the original excitement that got you here in the first place.

Of course, this is much easier said. This comes as naturally as eating nails. It feels like giving up. Of letting down everyone - your customers, your team, and your investors.

But take the other side of this argument. Say you continue pushing forward. The success of a startup is determined, in a large part, by the team behind it. The earlier and smaller the startup, the more of an outsized impact the founder will have in that team and hence the startup. Given the dismal odds of most startups succeeding, how likely will it be for your startup to do so if you're already burned out?

Continuing to push forward at this point is like the airport making people take off their shoes in the security line - playing along for appearance's sake vs actually addressing the real issues.

Sometimes the only way forward is to step back.

Are you burned out? If so, consider taking a break. The game will still be there when you get back. As long as you are still willing to play.

I used to have a sticker on my laptop that said "My other computer is Amazon EC2". This was true in the sense that I had many EC2 instances running at any given time doing compute on data and shuffling packets around the web. Amazon EC2 was not, however, a platform I used for day-to-day development.

My daily driver was a 13" Macbook. This worked well until it didn't. It started not working so well when memory became a bottleneck - this usually happened when I did anything with docker or had too many tabs open on Firefox.

After one too many computer panics due to memory issues, I invested in making EC2 my primary development environment. This transition happened around 2015 and I have been running on it ever since.

Running your dev environment on the cloud is a paradigm that has only become more versatile over time, improving from better instance types (eg. gravitron), development-friendly features (eg. hibernation), and developer tooling (eg. CDK).

You can find my dev setup on github as a CDK template. Executing the template will provision a personal cloud-based dev environment that can be spun up anywhere in the world (where there are AWS data centers).

Unfortunately, it's the year 2023 and setting this up via IaC (infrastructure as code) still requires some knowledge of AWS plumbing. This post goes over some of those technical details.

Architecture

NOTE: this article assumes you're familiar with the basics of AWS

If you deploy using the personal-cloud template, you'll end up with the following resources:

• 1 VPC

• 1 Autoscaling Group

• 1 EC2 Instance

VPC

The VPC consists of:

• 1 public subnet (has inbound and outbound paths to the internet)

• 0 private subnets (has no inbound but outbound to the internet)

• 1 isolated subnet (no path to public internet)

• 0 NAT gateways

Since I'm using this as a dev environment, the EC2 instance is launched in a public subnet. This is because we'll need to pull in packages from various programming languages (eg. npm) as well as download dependencies (eg. docker images) that we don't want to pay extra network fees for via a NAT gateway. The EC2 instance itself is locked down via security group rules that only permit inbound on port 22.

There are no private subnets as this stack is not meant to run any services besides development ec2 instances. I keep an isolated subnet around in reserve in case I need to spin up additional resources outside of my ec2 instance (eg. a database on a separate ec2 instance).

Note that the template by default creates a separate VPC. If you are working in a corporate environment, you might already have a development account and an existing VPC to launch your EC2 instance into. That might be preferable as you can then also connect to other internal services from inside the VPC.

Autoscaling Group

I like to add an autoscaling group with EC2, even if it is just a single EC2 instance. Autoscaling by itself does not cost anything and gives us health checks for free. If I do end up needing multiples of the same development instance, it will also be easy to provision by toggling the min-capacity threshold.

EC2 Instance

The EC2 instance has a custom security group that by default, only allows traffic from port 22 (ssh). You can optionally add additional inbound depending on your use case (eg. testing a WebSocket endpoint).

I generally find that a lot of work I do is either memory-bound or CPU bound. Depending on the case, I will provision a r6a or c6a type of ec2 instance.

To decode the instance family names:

• R: memory-optimized

• C: compute optimized

• A: AMD based (generally cheaper than i[ntel] based architectures)

NOTE: in most cases, you can also use arm-based gravitron instances (eg. swap out r6a with r6g) to deliver slightly better performance at less cost. Since most of my development work still targets non-arm architectures, I like to keep my environment the same as that of my target environments. If you end up using gravitron, you can use the newest generation c7g and r7g instances which offer the best "compute/memory to price-performance ratio" on AWS

The operating system is Amazon Linux 2. Note that this does not have many of the development tools you might be used to by default in other Linux distributions (eg. git). It is a secure Linux distro that is tuned to run with AWS and has long-term support.

I generally add some common user data scripts (a set of commands that run at the startup of a cloud VM) that add some developer tools like git and docker.

Note that by default, EBS-backed AWS instances will launch with 8GB of disk space. The template changes the default to 100GB.

Etc

The architecture here represents the basics of getting something running in AWS. There are many additional things you can do to optimize in terms of cost and DX.

Some of my favorites which are not covered in this post:

• adding tailscale to ec2 so you can use it to create your personal VPC of self-hosted things

• hibernating ec2 instance when not in use (and creating a script to automatically do this)

• connecting to your ec2 instance via vscode

• using reserved instances to further reduce costs by up to 75%

Conclusion

Today, it would be more accurate to simply say that "my computer is Amazon EC2". AWS gives you infinite freedom in creating a custom dev environment tailored for anything you might need. With this freedom also comes great complexity and hidden foot guns. The goal of this post is to give you a starting point for creating your personal cloud. Enjoy responsibly and watch out for dragons.

My main gripe with Firefox was its insatiable appetite for memory as it would regularly consume 10GB in the course of a day.

Enter Arc - a modern take on the web browser that's meant to be the "operating system for the web." Built on top of Chromium and advertised as a reimaging of what the browser could be, I recently got an invite to try it out. After two weeks of using it as my default browser, I'm not going back.

Arc

The first thing you'll notice about Arc is the sidebar. It's a jack-of-all-trades that functions as the address bar, menu, bookmarks, and an entire drawer full of additional functionality.

The sidebar itself exists as part of a "space" - a preconfigured set of tabs that is attached to a profile. Profile are like Firefox containers - they let you isolate online identities by compartmentalizing cookies and browser storage between different profiles.

With profiles, spaces, and tabs, Arc gives me the power features I needed from Firefox with the affordances of Chrome. It's compatible with all Chrome extensions (at least the ones I use), and its developer tools function as you'd expect from a Chromium browser. Additionally, Arc has many awesome quality-of-life features not found in other browsers, like the ability to do video in video when switching tabs and the ability to peek at links before committing to replacing the current page with it.

Going beyond the features, Arc is simply delightful to use. It is packed with little flourishes like musical icons that radiate from the tab when sound is playing and vibrant color palettes to make each space feel distinct.

Arc works so well that I've switched to using it for most of my Electron-based apps like Slack, Discord, and Notion. This ends up saving memory, but it does mean retraining my fingers to do Alt-Tab instead of Shift-Tab to switch between apps.

Parting Thoughts

Arc is a delightful browser developed by an amazing team (including one of the creators of google chrome). It's become my default browser over the last two weeks and I don't see myself going back.

The browser space has been in need of fresh air for some time, and Arc delivers just that and more.