about
About Me
- GITHUB: CalvinNeo
- EMAIL: calvinneo@calvinneo.com
- BLOG: www.calvinneo.com
- EDUCATION:
- BE: Civil Engineering at Hohai University
- ME: Software Engineering at Nanjing University
- HONOURS: I have won several prizes in ACM/ICPC, CCPC, RoboCup(China Region), Zhou Peiyuan Mechanics Competition, etc.
- OPEN-SOURCED PROJECTS:
- Nuft is an implementation of Raft protocol.
- ATP is a reliable transmission protocol built on UDP.
- CFortranTranslator is a line-by-line translator from Fortran77/90 to C++14.
- PolyClip clips millions of circles and lines with a polygon in seconds.
- Citation4Word is a word citation format for thesis.
- NuCut is a graph partitioning system.
- INTERNSHIP:
- Tencent Inc. IEG Group. Shenzhen, China. June 2018 - Sept 2018
- CAREER:
- Tencent Inc. IEG Group. Shenzhen/Shanghai, China. July 2019 - Aug 2021
- A platform to predict professional LOL game winning rate.
- An AI for crushing game(like CandyCrush) to simulate online player’s behavior.
- A platform to optimize performance of IEG’s maching learning system.
- Fuse execution of JAR/SQL components.
- Develop machine learning algorithm modules, including classification/clustering/model evaluation. Metadata management of the platform.
- Optimization suggestion by analysing Spark/Gc/Event logs.
- PingCAP. Remote/Shanghai, China. Aug 2021 -
- TiFlash Committer, TiFlash Proxy Maintainer and TiKV Reviewer
- Refactoring management of TiFlash replicas.
- Maintain the KVStore component.
- Write the committed data into the underlying columnar storage.
- A minimal raft fsm to manage uncommitted region data on TiFlash.
- Implement Learner Read and ReadIndex to support read with strong consistency.
- Implement Raft Snapshots to support bootstrapping peers from various heterogeneous data.
- Proactive(async) flushing in background. Reduce latency by 50%.
- Mitigated the negative impact on TiFlash write latency caused by explicit fsync points during Raft log application in raftstore.
- Maintain the TiFlash Proxy component.
- See Implementation of Proxy for more introduction of TiFlash Proxy. However, it roughly do the following:
- Join the TiKV cluster as a raft Learner and replicate data from raft Leader.
- Align the different persisting strategies between TiKV and TiFlash.
- Interact with TiFlash through FFI interfaces.
- Decouple TiFlash Proxy from TiKV
- See Implementation of Proxy for more introduction of TiFlash Proxy. However, it roughly do the following:
- Cloud native TiFlash.
- Fast scale out and recovery on S3. Scaling the second TiFlash node throughput 17x, mem 20%↓, cpu 96%↓.
- The solution is kind like implementing the “Follower Snapshot” in a raft learner instance. However, We implemented it like an optional plugin, so it doesn’t affect the main logic in Raft, and any failure could lead to a neaty fallback.
- It is optimized and tailored specifically for S3-compatible storage.
- Applying a snapshot now entails waiting for a sufficiently recent checkpoint on S3 and then applying its metadata.
- We designed a cache that is adaptive to the metadata access pattern, minimizing the number of requests to S3.
- Introduce a unified storage in TiFlash to store both the data from multiple engines, so that it can provide a consistent snapshot based on S3.
- Fast scale out and recovery on S3. Scaling the second TiFlash node throughput 17x, mem 20%↓, cpu 96%↓.
- Partitioned Raft KV, TiFlash part.
- Support TiFlash join a PRKV cluster as a non-PRKV learner.
- Large-scacle TXNs on TiFlash
- Support decoding and transforming big snapshots in parallel. It then contribute to a file-base large txn implementation which is provided on S3-based Serverless edition.
- For non-S3-based edition, we propose a transaction-committing model formally verified using TLA+. Such model is built based on Raft Log rather than file.
- A rate limiter to prevent OOM.
- O11y
- I am an expert in observing memory usage observation and handle memory errors. I have established multiple memory-observation solutions based on jeprof and eBPF to cover diverse deployment scenarios.
- TiFlash Committer, TiFlash Proxy Maintainer and TiKV Reviewer
- Tencent Inc. IEG Group. Shenzhen/Shanghai, China. July 2019 - Aug 2021
- CV:
- Please send an email to calvinneo@calvinneo.com, identify yourself and get my up-to-date CV.
- I have an extensive collection of wooden modules. Check 系列拼装模型攻略 to find how I built them.
- I play a wide range of sports, such as table tennis, archery, badminton. I love hiking and cycling. I have been cycling for over 5000KM+ during the last 3 yrs.
- I am fond of board games. My favorite board game includes: Gaia Project, ArkNova, Dune, Splendor(especially the pokemon edition), 11 nimmt, Rummikub, Arge of War, Las Vegas, CABO, Camel Cup, AZUL, PowWow, The Castles of Burgundy.