In the previous lesson, we learned the basics of Rust syntax. Now, let’s put that knowledge into practice by writing a small CLI tool. You might encounter some unfamiliar syntax, but don’t worry about understanding everything right away. Treat it like copying for now.

HTTPie

We’ll create a CLI tool to help us handle various tasks. Let’s take HTTPie as an example and see how to create a similar CLI tool using Rust. HTTPie is a Python-based, user-friendly command-line tool similar to cURL. It helps us diagnose HTTP services more effectively.

The image below shows a POST request sent using HTTPie. Compared to cURL, HTTPie has improved usability, including syntax highlighting for different types of information.

HTTPie Example

Think about how you would implement HTTPie using your favorite language. What libraries would you use? How many lines of code would it take? Now, think about how you would do it in Rust. With these thoughts in mind, let’s start building this tool with Rust! Our goal is to achieve this in about 200 lines of code.

Feature Analysis

To create a tool like HTTPie, we need to outline the main features to implement:

  1. Command-line parsing: Handle subcommands and various parameters, validate user input, and convert these inputs into parameters we can understand internally.
  2. Send an HTTP request: Based on the parsed parameters, send an HTTP request and get the response.
  3. User-friendly output: Display the response in a user-friendly manner.

This process can be seen in the diagram below:

HTTPie Workflow

Let’s look at the libraries needed to implement these features:

  • For command-line parsing, Rust offers several libraries. We’ll use the officially recommended clap.
  • For the HTTP client, we used reqwest in the previous lesson. We’ll continue with it, but this time we’ll try its asynchronous interface.
  • For formatted output, to make the output as readable as HTTPie’s Python version, we’ll use the colored library for colorful terminal output.
  • Additionally, we’ll need a few more libraries: anyhow for error handling, jsonxf for formatting JSON responses, mime for handling MIME types, and tokio for asynchronous processing.

CLI Handling

With a basic idea in mind, let’s create a project named httpie:

cargo new httpie
cd httpie

Open the project directory with VSCode and edit the Cargo.toml file to add the required dependencies.

[package]
name = "httpie"
version = "0.1.0"
edition = "2021"


[dependencies]
anyhow = "1.0.83"
clap = { version = "4.5.4", features = ["derive"] }
colored = "2.1.0"
jsonxf = "1.1.1"
mime = "0.3.17"
reqwest = { version = "0.12.4", features = ["json"] }
tokio = { version = "1.37.0", features = ["full"] }

First, let’s add the CLI-related code in main.rs:

use clap::Parser;

#[derive(Parser, Debug)]
#[clap(version = "1.0", author = "Garry")]
struct Opts {
    #[clap(subcommand)]
    subcmd: SubCommand,
}

#[derive(Parser, Debug)]
enum SubCommand {
    Get(Get),
    Post(Post),
}

#[derive(Parser, Debug)]
struct Get {
    url: String,
}

#[derive(Parser, Debug)]
struct Post {
    url: String,
    body: Vec<String>,
}

fn main() {
    let opts: Opts = Opts::parse();
    println!("{:?}", opts);
}

Here, we’re using macros provided by clap to simplify the CLI definition. These macros generate additional code to handle CLI parsing. With clap, we only need to describe what data the CLI will capture using a data structure T. Then, using T::parse(), we can parse various command-line arguments. The parse() function isn’t defined by us; it is auto-generated by #[derive(Parser)].

Currently, we have defined two subcommands. In Rust, subcommands can be defined using enums, and each subcommand’s parameters are defined by their respective data structures Get and Post.

Let’s run this code:

cargo build --quiet && target/debug/httpie post httpbin.org/post a=1 b=2

Output:

Opts { subcmd: Post(Post { url: "httpbin.org/post", body: ["a=1", "b=2"] }) }

By default, the binary compiled by cargo build is located in the project’s target/debug directory. As we can see, the command-line parsing works, achieving the desired functionality.

Adding Validation

Currently, we haven’t added any input validation. For example, the following input results in an incorrectly parsed URL:

cargo build --quiet && target/debug/httpie post a=1 b=2

Output:

Opts { subcmd: Post(Post { url: "a=1", body: ["b=2"] }) }

We need to add validation. There are two validations to implement: one for the URL and another for the body.

First, Validate that the URL is Valid

use anyhow::Result;
use reqwest::Url;

#[derive(Args, Debug)]
struct Get {
    #[arg(value_parser = parse_url)]
    url: String
}

fn parse_url(s: &str) -> Result<String> {
    let _url: Url = s.parse()?;
    Ok(s.into())
}

clap allows you to add custom parsing functions for each value it parses. Here, we’ll define a parse_url function to check the validity of the URL.

Then, we need to ensure that each item in the body is in the key=value format. We can define a data structure KvPair to store this information and also define a custom parsing function to parse the results into KvPair.

use std::str::FromStr;
use anyhow::{anyhow, Result};

#[derive(Args, Debug)]
struct Post {
    #[arg(value_parser = parse_url)]
    url: String,
    #[arg(value_parser = parse_kv_pair)]
    body: Vec<KvPair>,
}


#[derive(Debug,PartialEq,Clone)]
struct KvPair {
    k: String,
    v: String,
}

impl FromStr for KvPair {
    type Err = anyhow::Error;

    fn from_str(s: &str) -> Result<Self,Self::Err> {
        let mut split = s.split("=");
        let err = || anyhow!(format!("Failed to parse {}", s));
        Ok(Self {
            k: (split.next().ok_or_else(err)?).to_string(),
            v: (split.next().ok_or_else(err)?).to_string(),
        })
    }
}

fn parse_kv_pair(s: &str) -> Result<KvPair> {
    Ok(s.parse()?)
}

Here, we implemented a FromStr trait to convert strings that meet the conditions into KvPair. The FromStr trait is defined by the Rust standard library, and once implemented, you can conveniently use the parse() generic function to handle the conversion of strings into a specific type.

After making these changes, our CLI is more robust. We can test it again.

❯ cargo build --quiet
❯ target/debug/httpie post https://httpbin.org/post a=1 b
error: Invalid value for '<BODY>...': Failed to parse b

For more information try --help
❯ target/debug/httpie post abc a=1
error: Invalid value for '<URL>': relative URL without a base

For more information try --help

target/debug/httpie post https://httpbin.org/post a=1 b=2
Opts { subcmd: Post(Post { url: "https://httpbin.org/post", body: [KvPair { k: "a", v: "1" }, KvPair { k: "b", v: "2" }] }) }

Cool, we have completed the basic validation. Notably, we didn’t clutter the main workflow with various validation code but instead used additional validation functions and traits to achieve this. These newly added pieces of code are highly reusable and independent of each other, and they do not require modifying the main workflow.

This approach aligns well with the Open-Closed Principle of software development: Rust, through its macros, traits, generic functions, and trait objects, helps us write well-structured and maintainable code more easily.

You might not fully understand all the details of this code right now, but don’t worry. Keep writing, and for today, just focus on getting the code to run without worrying about grasping every concept.

HTTP Request

Next, let’s continue with the core functionality of HTTPie: handling HTTP requests. We’ll add the subcommand processing workflow in the main() function.

use reqwest::{header, Client, Response, Url};

async fn main() -> Result<()> {
    let opts: Opts = Opts::parse();

    let client = Client::new();
    let result= match opts.command {
        SubCommand::Get(ref args) => get(client, args).await?,
        SubCommand::Post(ref args ) => post(client, args).await?,
    };

    Ok(result)
}

Notice that we turn the main function into an async fn, which represents an asynchronous function. For an async main, we need to use the #[tokio::main] macro to automatically handle the asynchronous runtime.

Inside the main function, we call the get and post functions to handle specific tasks based on the subcommand type. These functions are implemented as follows:

use std::{collections::HashMap, str::FromStr};

async fn get(client: Client, args: &Get) -> Result<()> {
    let resp = client.get(&args.url).send().await?;
    Ok(print_resp(resp).await?)
}

async fn post(client: Client, args: &Post) -> Result<()> {
    let mut body = HashMap::new();
    for kv in args.body.iter() {
        body.insert(&kv.k, &kv.v);
    }
    let resp = client.post(&args.url).json(&body).send().await?;
    Ok(print_resp(resp).await?)
}

In these implementations, the parsed list of KvPair needs to be placed into a HashMap and then passed to the HTTP client’s JSON method. This completes the basic functionality of our HTTPie.

Currently, the printed data is not very user-friendly. We need to further print the HTTP headers and HTTP body in different colors, similar to the Python version of HTTPie. This part of the code is relatively simple, so we won’t go into detail here.

Finally, let’s look at the complete code.

use clap::{Args, Parser, Subcommand};
use anyhow::{Result, anyhow};
use reqwest::{header, Client, Response, Url};
use std::str::FromStr;
use std::collections::HashMap;
use colored::*;
use mime::Mime;

#[derive(Parser, Debug)]
#[clap(version = "1.0", author = "Garry")]
struct Opts {
    #[command(subcommand)]
    command: SubCommand,

}

#[derive(Subcommand, Debug)]
enum SubCommand {
    Get(Get),
    Post(Post),
}

#[derive(Args, Debug)]
struct Get {
    #[arg(value_parser = parse_url)]
    url: String
}

fn parse_url(s: &str) -> Result<String> {
    let _url: Url = s.parse()?;
    Ok(s.into())
}

#[derive(Args, Debug)]
struct Post {
    #[arg(value_parser = parse_url)]
    url: String,
    #[arg(value_parser = parse_kv_pair)]
    body: Vec<KvPair>,
}


#[derive(Debug,PartialEq,Clone)]
struct KvPair {
    k: String,
    v: String,
}

impl FromStr for KvPair {
    type Err = anyhow::Error;

    fn from_str(s: &str) -> Result<Self,Self::Err> {
        let mut split = s.split("=");
        let err = || anyhow!(format!("Failed to parse {}", s));
        Ok(Self {
            k: (split.next().ok_or_else(err)?).to_string(),
            v: (split.next().ok_or_else(err)?).to_string(),
        })
    }
}

fn parse_kv_pair(s: &str) -> Result<KvPair> {
    Ok(s.parse()?)
}

async fn get(client: Client, args: &Get) -> Result<()> {
    let resp = client.get(&args.url).send().await?;
    Ok(print_resp(resp).await?)
}

async fn post(client: Client, args: &Post) -> Result<()> {
    let mut body = HashMap::new();
    for kv in args.body.iter() {
        body.insert(&kv.k, &kv.v);
    }
    let resp = client.post(&args.url).json(&body).send().await?;
    Ok(print_resp(resp).await?)
}

fn print_status(resp: &Response) {
    let status = format!("{:?} {}", resp.version(), resp.status()).blue();
    println!("{}\n", status);
}

fn print_headers(resp: &Response) {
    for (name, value) in resp.headers() {
        println!("{}: {:?}", name.to_string().green(), value);
    }
    println!();
}

fn print_body(m: Option<Mime>, body: &String) {
    match m {
        Some(v) if v == mime::APPLICATION_JSON => {
            println!("{}", jsonxf::pretty_print(body).unwrap().cyan());
        }
        _ => println!("{}", body),
    }
}

fn get_content_type(resp: &Response) -> Option<Mime> {
    resp.headers()
        .get(header::CONTENT_TYPE)
        .map(|v| v.to_str().unwrap().parse().unwrap())
}

async fn print_resp(resp: Response) -> Result<()> {
    print_status(&resp);
    print_headers(&resp);
    let mime = get_content_type(&resp);
    let body = resp.text().await?;
    print_body(mime, &body);
    Ok(())
}


#[tokio::main]
async fn main() -> Result<()> {
    let opts: Opts = Opts::parse();
    let mut headers = header::HeaderMap::new();
    headers.insert("X-POWERED-BY", "Rust".parse()?);
    headers.insert(header::USER_AGENT, "Rust Httpie".parse()?);
    let client = reqwest::Client::builder()
        .default_headers(headers)
        .build()?;
    let result= match opts.command {
        SubCommand::Get(ref args) => get(client, args).await?,
        SubCommand::Post(ref args ) => post(client, args).await?,
    };

    Ok(result)
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]    
    fn parse_url_works() {
        assert!(parse_url("abc").is_err());        
        assert!(parse_url("http://abc.xyz").is_ok());        
        assert!(parse_url("https://httpbin.org/post").is_ok());    
}    
    #[test]    
    fn parse_kv_pair_works() {        
        assert!(parse_kv_pair("a").is_err());        
        assert_eq!(            
            parse_kv_pair("a=1").unwrap(),            
            KvPair {                
                k: "a".into(),                
                v: "1".into()            
            }        
        );        
        
        assert_eq!(            
            parse_kv_pair("b=").unwrap(),            
            KvPair {                
                k: "b".into(),                
                v: "".into()            
            }        
        );    
    }
}

At the end of this complete code, I also wrote a few unit tests that you can run with cargo test. Rust supports conditional compilation, and the #[cfg(test)] attribute indicates that the entire mod tests will only be compiled during cargo test.

Using the line count tool tokei, we can see that we used a total of 139 lines of code to achieve this functionality, including about 30 lines of unit test code:

> tokei src/main.rs
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1          163          139            0           24
===============================================================================
 Total                   1          163          139            0           24
===============================================================================

You can compile the release version with cargo build --release, copy it to a directory in your $PATH, and then try it out.

At this point, a fully functional HTTPie with complete help is ready for use.

Let’s test the result.

result

Ha, we succeeded with this example. We only used just over 100 lines of code to implement the core functionality of HTTPie, which is well below the expected 200 lines. I hope you can vaguely sense Rust’s ability to solve real-world problems. Using today’s implementation of HTTPie as an example:

  • To parse the command line into a data structure, we just needed to add some simple annotations to the data structure.
  • Data validation can be handled by separate functions that have no coupling with the main workflow.
  • As a CLI parsing library, clap’s overall experience is very similar to Python’s click, but simpler than Golang’s cobra.

This demonstrates the power of the Rust language. Despite being designed for system-level development, it can provide abstractions and experiences similar to Python. Once you get used to Rust, using it feels remarkably pleasant.