engineering: visualizing crypto market outperform BTC-Alt indicators …

…with Golang

AI/Building LLM system/guardrails-in-llm.md

@@ -16,7 +16,7 @@ Guardrails in LLM are a set of techniques and strategies designed to control and
 
 ## Types of guardrails
 
-![Guardrails in LLM](./assets/guardrails-in-llm.webp)
+![Guardrails in LLM](assets/guardrails-in-llm.webp)
 
 1. **Input guardrails**: This involves pre-processing the input to the model to remove or modify any potentially harmful or inappropriate content. This can include filtering out profanity, hate speech, or sensitive information. Some common usecases:
    - **Topical guardrails**: Limit the model's responses to a specific topic or domain to prevent it from generating off-topic or irrelevant content.

AI/Building LLM system/quantization-in-llm.md

@@ -40,11 +40,11 @@ There are many quantization schema to reduce the size of the model. One techniqu
 
 In this part, we focus on the asymmetric mode.
 
-![Asymmetric mode](./assets/quantization-in-llm-formula.webp)
+![Asymmetric mode](assets/quantization-in-llm-formula.webp)
 
 In this part, we focus on the asymmetric mode.
 
-![Asymmetric mode](./assets/quantization-in-llm-formula.webp)
+![Asymmetric mode](assets/quantization-in-llm-formula.webp)
 
 The fundamental formula is:
 
@@ -132,4 +132,4 @@ Quantization stands as a pivotal technique in LLM optimization, enabling efficie
 * https://medium.com/@lmpo/understanding-model-quantization-for-llms-1573490d44ad
 * https://www.datacamp.com/tutorial/quantization-for-large-language-models
 * https://medium.com/@vimalkansal/understanding-the-gguf-format-a-comprehensive-guide-67de48848256
-* https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization
+* https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization

Use Cases/binance-transfer-matching.md

@@ -1,5 +1,5 @@
 ---
-title: "Building Better Crypto Transfer Tracking"
+title: "Building better Binance transfer tracking"
 date: 2024-11-18
 tags:
   - data
@@ -48,12 +48,12 @@ To record every transfers, we need the help of Binance APIs, specifically is [Ge
 
 Our job is just passing `TRANSFER` as `incomeType` to filter out other types of Binance transactions. Then we can store these records for use later. But when looking at this response, can you imagine the limitations that I mentioned in the title of this part? Yes! you actually can't know where the fund comes from or move to? Just only can detect whether it is a deposit or withdrawal by using the sign, which is not enough in our system where every account is under our control. If it is hard for you to understand, the result of the transfer notification is look sus as below screenshot.
 
-![Sporadic and confusing transfer logs](./assets/nghenhan-bad-logging.png)
+![Sporadic and confusing transfer logs](assets/nghenhan-bad-logging.png)
 *Figure 1: Sporadic and confusing transfer logs that lack clear relationships between transactions*
 
 To me, it looks bad. Ignore the wrong destination balance because of another issue with the data, this logging is too sporadic, hard to understand, and confusing. We can't understand how the fund is transferred. In my expectation, at least, it should like following.
 
-![Clear and connected transfer logs showing fund flow between accounts](./assets/nghenhan-better-logging.png)
+![Clear and connected transfer logs showing fund flow between accounts](assets/nghenhan-better-logging.png)
 *Figure 2: Clear and connected transfer logs that show the complete flow of funds between accounts*
 
 If you pay attention to the `JSON` response of Binance API, an idea can be raised in your mind that "*Hmm, it looks easy to get the better version of logging by just only matching the transaction ID aka tranId field value*". Yes, it is the first thing that popped into my mind. Unfortunately, once the transfer happens between two accounts, different transaction IDs are produced on each account side.

Use Cases/crypto-market-outperform-chart-rendering.md

@@ -0,0 +1,140 @@
+---
+title: "Visualizing crypto market outperform BTC-Alt indicators with Golang"
+date: 2024-11-18
+tags:
+  - data
+  - crypto
+  - golang
+description: "Implementing a Golang-based visualization for crypto market performance indicators, focusing on Bitcoin vs Altcoin dynamics and trading strategy effectiveness through interactive charts and data analysis"
+authors:
+  - bievh
+---
+
+Crypto trading is not just gambling. It has strategies. Once traders can take advantage of it effectively, they can earn. One of the most popular strategies in cryptocurrency is **Hedge**, which is often mentioned with the name **Hedge Bitcoin**.
+
+In simple words, assuming that we are living in a world where we can only just buy/sell salt and pepper. As a merchant, you buy one for speculation. But the risk happens when the price of your holding asset decreases; you lose. With the **Hedge** strategy, you buy both of them with the belief that if one of them decreases, the other will increase. For example, if the price of salt increases while the price of pepper decreases, you buy more pepper and sell a part of your salt.
+
+The real strategy is more complex than my example. But we can understand that **Hedge Bitcoin** means trading entirely other assets on the exchange in the inverse direction to decrease the risk of Bitcoin trading. **To execute this strategy effectively, traders must monitor how liquidity flows between Bitcoin and Altcoins across the exchange, revealing their relative performance patterns.**
+
+### How can we extract market-wide insights from trading data
+
+Performance, in our context, represents the token's profit and loss. For example, if I say, *Performance of BTC is 10%,* it means BTC has profited 10% since we placed the order. This number closely relates to the asset's liquidity in the market. If you can't imagine the relationship between them, try thinking about the price first. The price of an asset only increases when its total liquidity in the market rises. The change can come from scenarios such as the following:
+- Liquidity is moved from other assets to this asset
+- Liquidity from new buyers entering the market 
+
+![alt text](assets/liq-price-pnl-relationship.png)  
+*Figure 1: Relationship between asset liquidity, price, and trading PnL*
+
+Price movements serve as indicators of liquidity flows between assets. When capital moves between Bitcoin and Altcoins, it triggers price changes that result in trading profits or losses. This chain reaction works in reverse too. Trading PnL patterns reveal price movements, which in turn expose the underlying liquidity shifts in the market.
+
+Finally, market performance can be calculated using the following formula:
+
+```math
+market_{perf} = BTC_{perf} - Alt_{perf} = \frac{BTC_{pnl}}{BTC_{init}} - \frac{\sum {Alt_{pnl}}}{\sum{Alt_{init}}} 
+```
+*Formula 1: Formula to calculate market performance*
+
+This formula serves as a powerful indicator, revealing liquidity movements across the market whether capital is flowing from Bitcoin to Altcoins, from Altcoins to Bitcoin, or entering/exiting the market entirely. Unlike evaluating price, which provides a general view of trends, performance allows us to quantitatively assess the market. For example:
+- When market performance is 5% and BTC performance is 2.5%, this means 2.5% of the funds are moved from Altcoins to BTC, and no new funds have entered the market.
+- When market performance is -10% and BTC performance remains unchanged, this means 10% of funds are poured into the market from outside.
+
+Now we have our framework to hedge. How can we obtain this data? Based on *Formula 1*, we need the trading PnL of BTC and all Altcoins. So our business is to prepare a Binance account; depositing money into it; opening a BTC long position with 50% of our funds, and splitting the rest 50% among short positions on all Altcoins. Finally, we wait for PnL changes at each interval, usually a minute; calculate the performances from the PnLs, and record them in our database for later use.
+
+### From data to dashboard: implementing a Go web interface to render performance charts
+At the end of the previous part, we mentioned the data gathering. Let me show you how was it constructed.
+
+![alt text](assets/market-perf.png)
+*Figure 2: Market performance data*
+
+In *Figure 2*, you can easily see that a large amount of liquidity has been pumped into the market. So, how easy is it to get an overview of the market when these figures are visualized? This is the reason why we are trying to do it here, until now. 
+
+Market performance alone is not enough. In addition to estimating future market trends for trading, we also need to evaluate the performance of past trades. To achieve this, I have integrated our trading history into the charts. These include:  
+
+- **Trade round**: The time period from the beginning to the end of each trade.  
+- **Trading account PnL history**: The PnL changes of a specific Binance account over time.  
+
+By visualizing both market trends and historical trading data, we gain a more comprehensive understanding of our trading efficiency and decision-making process. The first step is selecting the most appropriate type of chart. It must ensure that when multiple data sources are combined, the visualization retains clarity, readability, and meaning. A mixed chart is ideal for this purpose, combining lines to represent market performance and trading PnL changes over time with a double bar chart that juxtaposes BTC and Altcoin performance to highlight their variations. Finally, scoping each trade within a separate window allows us to analyze individual trading periods in detail.
+
+![alt text](assets/perf-chart.png)
+*Figure 3: The mixed chart that represent the relationship between market performance and trading effective*
+
+#### Aggregating lines from multiple sources
+We will begin the implementation with aggregating and aligning data from different source. The performance data, round period and PnL records each come with their own structure and time frames. To ensure everything aligns properly on the time axis, the data is mapped using time-based Golang map as following.
+
+```go
+performanceMap := make(map[string]Performance)
+for _, perf := range performances {
+    performanceMap[perf.Time.Format("2006-01-02 15:04:05")] = perf
+}
+
+pnlMap := make(map[string]Pnl)
+for _, pnl := range pnls {
+    pnlMap[pnl.Time.Format("2006-01-02 15:04:05")] = pnl
+}
+```
+*Code 1: Simple mapping value of performance and pnl to the time axis*
+
+#### Construct the chart
+```go
+line := charts.NewLine()
+line.SetGlobalOptions(
+    charts.WithTitleOpts(opts.Title{Title: "Performance and PnL"}),
+    charts.WithXAxisOpts(opts.XAxis{Name: "Time"}),
+    charts.WithYAxisOpts(opts.YAxis{
+        Name: "Performance (%)",
+        Min: -maxPerf,
+        Max: maxPerf,
+    }),
+)
+
+line.ExtendYAxis(opts.YAxis{
+	Name: "PnL ($)",
+	AxisLabel: &opts.AxisLabel{
+		Formatter: "{value} $",
+	},
+	Min: -maxPnL,
+	Max: maxPnL,
+	AxisLine: &opts.AxisLine{
+		Show:            &trueval,
+		OnZeroAxisIndex: 1,
+	},
+})
+
+line.AddSeries("Performance", yAxisPerf)
+line.AddSeries("Unrealized PnL", yAxisPnl, charts.WithLineChartOpts(opts.LineChart{YAxisIndex: 1}))
+
+bar := charts.NewBar()
+bar.SetXAxis(xAxis).
+    AddSeries("AltCoin Perf", yAxisShortPerf).
+    AddSeries("BTC Perf", yAxisLongPerf)
+
+line.Overlap(bar)
+```
+*Code 2: Code snippet demonstrates how to create a mixed chart using the **go-echarts** library*
+
+To combine line and bar charts to visualize market performance, PnL, and BTC/Altcoin performance. The process begins with defining a line chart using the `charts.NewLine()` function. This line chart is configured with global options such as the title, X-axis for time, and a Y-axis labeled "Performance (%)", which ranges from `-maxPerf` to `maxPerf`. This setup ensures that the performance data is plotted on a dedicated axis, making it easy to interpret trends over time.
+
+Next, a secondary Y-axis is added to represent PnL, labeled "PnL ($)". This axis is configured with its own range (`-maxPnL` to `maxPnL`) and includes an axis line centered on zero for better visual balance. By extending the Y-axis with this configuration, the chart supports two distinct datasets on different scales, ensuring both performance and PnL are displayed clearly without visual clutter.
+
+The `line.AddSeries` method is used to add the performance data and unrealized PnL data to the chart. Each dataset is represented as a separate line, with the PnL data assigned to the secondary Y-axis using `charts.WithLineChartOpts(opts.LineChart{YAxisIndex: 1})`. This approach ensures that performance and PnL are plotted on their respective axes, maintaining the clarity of the visualization.
+
+To include BTC and Altcoin performance data, a bar chart is created using `charts.NewBar()`. This bar chart shares the same X-axis as the line chart and is populated with series for "AltCoin Perf" and "BTC Perf" using the `AddSeries` method. The bar chart highlights how the performance of these two asset classes changes over time, complementing the overall visualization.
+
+The `line.Overlap(bar)` method combines the line and bar charts into a single cohesive visualization. This allows the user to analyze market trends, PnL changes, and asset performance simultaneously within one chart.
+
+After all, we may still missing something. Yes it is the area to represent the boundaries of trade rounds. It is a bit easy, **go-echart** provides us the option `charts.WithMarkAreaNameCoordItemOpts` to integrate the mark areas to the line chart. Our business is construct each area boundaries by specify its coordinates using trade round start, end time.
+```go
+opts.MarkAreaNameCoordItem{
+				Coordinate0: []interface{}{trade.OpenedTime.Format("2006-01-02 15:04:05"), -maxYAxis},
+				Coordinate1: []interface{}{trade.ClosedTime.Format("2006-01-02 15:04:05"), maxYAxis}, 
+				ItemStyle: &opts.ItemStyle{
+					Color: "rgba(255, 255, 255, 0.3)", // White color with blur effect
+				},
+			},
+```
+*Code 3: Code snippet to construct MarkAreaNameCoordItem depend on trade period*
+
+### Conclusions
+This project demonstrates how we can combine Golang and go-echarts to build powerful visualizations that provide deep insights into crypto trading performance. By integrating market trends, historical PnL data, and trading rounds into a single chart, we create a tool that allows traders to make informed decisions with clarity and precision.
+
+The challenges of aligning multiple datasets, ensuring readability, and maintaining meaningfulness were overcome with careful design and thoughtful implementation. This visualization not only simplifies the analysis of complex trading data but also empowers users to refine their strategies and improve efficiency. As the crypto market continues to evolve, tools like these will be indispensable for staying ahead of the curve.

devbox/introduction/The reason for being.md

@@ -16,14 +16,14 @@ With the rise of cloud services, we first adapted Docker as a containerization t
 ### Bad build cache
 Docker build is actually not very effective because basically, we need to rebuild the entire environment each time our code is changed when developing. This rebuild process not only affects the image layer where the code is placed but also lets other layer below it is rebuilt. It takes more time and resource to do it.
 
-![alt text](./assets/docker_layers.png)
+![alt text](assets/docker_layers.png)
 
 ### Pulling from internet is not stable
 The installed application version can be specified in the `apt-get add` script, but its dependencies are not stable. Repositories can be updated, transitive dependencies are changed without changing the main package version. So once we run `apt-get update`, the produced image may not be the same as the previous build.
 
 ### Docker is heavy on non-Linux OS
 On Windows and MacOS, Docker containers are run on a Linux VM. So we need to split resources from the host machine to serve this VM. It makes our computer become slow.
-![alt text](./assets/docker_container_run.png)
+![alt text](assets/docker_container_run.png)
 
 ### How to resolve problems?
 The problems can be summed up as slow, asynchronous, and cumbersome when developing locally. We actually can resolve them by using some Docker hack. But we can also do it by avoiding using Docker (or changing your machine if you want). Among a variety solutions, we consider trying using Devbox. What's it?
@@ -36,7 +36,7 @@ Powered by Nix which is known as a cross-platform package manager for Unix-like
 ### Consistent dependencies
 All packages in the Devbox workspace are actually linked from your local Nix storage, where every application is installed. They are identified by a unique hash and are linked together by a dependency tree. So it will not change over time. So the Devbox environment can be reproduced perfectly and easily to share between teammates with just a script file.
 
-![alt text](./assets/nix_deps_graph.png)
+![alt text](assets/nix_deps_graph.png)
 
 The hash is different between versions of the same application. So we have different versions of a package in our Nix storage. And easily link it to different projects that require different versions of a package.