Cloudyy - Blog

Pi-Hole instance on the cloud & security

CloudyyUw — Sun, 02 Mar 2025 14:39:57 GMT

Backstory

Since I decided to sell my old Dell Optiplex that I used to self host things, the only service that I kept hosting at home was a Pi-Hole instance on an old phone.

It works, sort of.. Like it worked as expected after I found this project, but even for something as well optimized and made to run on a Raspberry Pi, it was horrendously slow. But it was kind of ok, not very reliable or fast, but it got the job done at the end of the day..

But there's a little problem with hosting things at home: when you're not at home. I know that something like ZeroTier could solve this, especially since the mobile client has the option to set custom DNS addresses. I was just not brave enough to create a bomb, since the phone could barely run the Pi-Hole instance, adding something like that could potentially create a fire hazard.

So I decided to host it in the cloud, solving both speed and external access problems. At this point I was well aware that doing this was a terrible idea since your instance could be a public resolver, but with DNSSEC and rate limiting I thought it was okay.

Then, very quickly, the thing that ruins the internet began to show up: bots. Some were just friendly security research bots that just ping the server, others were not clear about their purpose and multiple instances of them kept pinging, and then there were the Chinese ones that actually pissed me off a bit.

It's not that I want to generalize, but every single one of the annoying bots had a Chinese IP address. The majority made a bunch of queries from a handful of different IP addresses, quite a few from the same CIDR.

Now that I think about it, one of them had a kind of natural use, as if a person was actually using it, which makes me think if someone was actually using it...

Anyway, back to the Chinese bots. One day I went to do the usual check on the instance, everything was fine until I looked at the client list. A new bot had made about 500k queries, turning on the query log (which I had turned off since the log file size skyrockets with DNSSEC), it was slamming the service with hundreds of bogus domains, just different enough from each other to not fall into the Pi-Hole cache.

Actually, congratulations to the Pi-Hole development team for making it so optimized. If I hadn't checked the client list, I wouldn't have noticed a difference as it handled those hundreds of queries per minute like it was nothing.

And what about now? How do we block all queries from unknown clients without too much work?

The solution was behind a neglected feature that I never thought to use: groups. This way I could create a blocked group that all new clients would fall into, and a whitelisted group with all my clients.

Meet `(.*?)`

By adding this regex pattern to your domain filter and regex denying it on the Default group, all queries from all clients will be blocked.

Just make sure you don't enable it before creating a whitelisted group and adding yourself to it, otherwise you may end up blocking yourself as well. Yes, I'm talking from experience 🥲.

Then all you need to do is create another group where this rule doesn't apply, add your clients to it, and you've successfully created a whitelist.

Now you can sit back and relax, right?

IP addresses change

IP addresses change over time and across networks, making manual whitelisting a pain, especially since you are automatically blocked if you are on a different client/IP address.

The laziest solution would be to whitelist your *CIDR, since at least in my experience, just doing that handles any eventual small changes well.

*: Imagine this is your public IP address: 192.123.45.67, the CIDR would be 192.123.45.0/24.

But what about if you are on mobile data or on public wi-fi? These IPs change all the time and in mobile data, just allowing a CIDR will not do the trick because the whole IP changes frequently.

Engineers solve practical problems

Keeping track of changing addresses... doesn't that sound like a task that could be automated? If your answer was yes, it really is! If it was no... umm, isn't this post extremely boring for you?

So I decided to write a simple Bash script just as a proof of concept, I was then shot 57 times.

Sorry, that meme was the first way I thought of expressing how I felt about looking for documentation for the Pi-Hole API for longer than it should be, only to find out a day later that it lives inside your instance... Why? I don't know, maybe they wanted to take the Forgejo approach and make it local and interactive, which is good to be able to test things directly in your browser, but Forgejo makes it clean with a link in the footer.

Anyway, finally with the documentation, I decided to code it in Rust. Yes, at this point I gave up on doing it in Bash, since it would actually be easier to implement more complex stuff and make it more portable with Rust.

The steps are actually quite simple:

Authenticate with the Pi-Hole instance.
Find the ID of the allowed group.
Get your public IP address.
Put yourself in the whitelist.
~~Read the compiler's death threats about your bad code, optional.~~

I'll do my best to keep the explanation as simple and short as possible. Also don't mind if the code is bad, it's been a while since I used Rust, you could say my skills are kind of... you know the joke 😐.

Storing some configs

Before getting started, it's good to organize what variables our program will use. For this use case we need to store:

The app password for authentication
The URL of the API
The name of the whitelisted group

With that in mind, let's create a conf.toml file with the following contents:

api_url = "https://pihole.example.org/api"
app_password = "secret password"
whitelisted_group_name = "whitelisted group name"

And to be able to load this config file into the program, let's create a simple wrapper function:

use serde::Deserialize;

#[derive(Deserialize)]
pub struct Config {
    pub api_url: String,
    pub app_password: String,
    pub whitelisted_group_name: String,
}

fn config() -> Config {
    let conf_file = std::fs::read_to_string("./conf.toml").expect("Failed to read conf.toml");

    let parsed: Config =
        toml::from_str(&conf_file).expect("Failed to parse conf.toml");

    parsed
}

With that properly set up, we can jump right into the first step of the process: authenticate with the Pi-Hole API.

Authentication with the API

Pi-Hole has a pretty straightforward approach to auth, POST your app password to /api/auth and get an SID. So, based on our config:

use reqwest;

async fn auth() {
	let body = serde_json::json!({
		"password": config.app_password
	});
	let res = reqwest::Client::new()
		.post(format!("{}/auth", conf.api_url))
		.json(&body)
		.send()
		.await
		.unwrap();
	println!("{:?}", res.text().await);
}

Then in your console you should see something like:

{
	"session": {
		"valid": true,
		"sid": ".......",
		...
	}
	...
}

If you didn't got any errors, you are all good to handle the response and proceed to step 2: fetching the group ID.

Group IDs

In order to add ourselves to the correct group, we need to know which is the correct ID of that group. In reality, it's quite easy to know, since the Default group has an ID of 1. But just knowing the answer is no fun, and there's probably someone with a dozen groups, so let's use the API for that.

With our SID in hands, we could call /api/groups/[group name]:

async fn fetch_group_id(sid: String) {
	let mut headers = header::HeaderMap::new();

	headers.insert("X-FTL-SID", sid.parse().unwrap());

	let res = reqwest::Client::new()
		.get(format!(
			"{}/groups/{}",
			conf.api_url, conf.whitelisted_group_name
		))
		.headers(headers)
		.send()
		.await
		.unwrap();
	println!("{:?}", res.text().await);
}

Output:

{
	"groups": [
		{
			"name": "Whitelist",
			"id": 2
		}
	]
	...
}

No errors? Awesome! Now to the next step: find your IP address.

What is my IP address

Despite the title, we will not use any of these websites for this example. Instead, we will use an instance of traefik/whoami. Why do this? I just had one running for some testing and since it already has an API, why not?

Actually, since this step might be different for your use case, I'm going to assume that you already have your public IPv4 address, and now you're just getting the CIDR. This step actually took me a minute before I realized "wait, I just want to put a zero and a forward slash, just split the string bro".

And after resisting the urge to search "How to split a string in Rust" on StackOverflow, the code is:

let ip_parts: Vec<&str> = ip_addr.split('.').collect();

if ip_parts.len() != 4 {
	println!("Sir that's not a valid IPv4 address");
}

let cidr = format!("{}.{}.{}.0/24", ip_parts[0], ip_parts[1], ip_parts[2]);

And with that we've completed all the requirement steps, just one more to go.

Whitelisting the client

Just as simple as the other steps: POST to /api/clients with some info:

async fn whitelist_client(sid: String, group: i32, client_cidr: String) {
	let mut headers = header::HeaderMap::new();
	let body = serde_json::json!({
		"client": client_cidr,
		"comment": "Comment",
		"groups": [group]
	});

	headers.insert("X-FTL-SID", sid.parse().unwrap());

	let res = reqwest::Client::new()
		.post(format!("{}/clients", conf.api_url))
		.json(&body)
		.headers(headers)
		.send()
		.await
		.unwrap();
	println!("{:?}", res.text().await)
}

If that was successful, we are done! This endpoint likes to "fail with grace" by returning a 200 but error-ed, this usually happens when the client is not unique, but it's no big deal.

What if I'm already blocked?

Let's say you did a little opsie, blocked yourself, and your Pi-Hole instance is only accessible through a reverse proxy (a situation that has never happened to me, of course), how about now?

Well, if you are blocked, it means that you are not able to resolve any domain name (unless you have it set up in /etc/hosts), but you still have internet access. This means we can use a little (unsafe) trick: just contact the server directly by it's IP address.

To do that with cURL it's pretty simple:

curl --request GET \
	--url https://<ip>/api \
	--header 'Host: pihole.example.org' \
	-k

Since we are explicitly specifying the Host header, TLS will freak out and fail. So we need to ignore it by using the -k flag. This is of course insecure, but it solves our problem.

To implement it with reqwest we will just need to create a client with this option:

let req_client = reqwest::Client::builder()
	.danger_accept_invalid_certs(true)
	.build()
	.unwrap();

Now, by replacing the usual Client::new() with this client, it will be possible to ignore TLS errors. (Did I mentioned that this is insecure?)

Then you could pass the host with:

use reqwest::header;

let mut headers = header::HeaderMap::new();
headers.insert(header::HOST, "pihole.example.org".parse.unwrap())

Conclusion

Well, this is it. It was pretty fun to come up with this excuse to do some coding but solve this persistent problem. Hopefully this will be useful to someone else.

The full code is also available on my Codeberg profile if you want to check out everything.

See ya!

Using Telegram as infinite cloud storage

CloudyyUw — Wed, 14 Feb 2024 20:51:59 GMT

17/08/24 Edit: I decided to rewrite this post because it was very technical and difficult to understand.

A while ago I was backing up my homelab and the total size was about 300 GB, which was no problem to store locally, but I still needed an external backup. So I started looking for a cloud storage provider, but they are too expensive for sporadic things like huge backups, and there is also the issue of trust.

While researching, I remembered a project a friend told me about a long time ago called Teledrive, a project for using Telegram to store files. But this project is dead and I was felling like doing some tinkering

How that works?

Telegram has 2 APIs, one for bots and one for clients. The bot API is usually easier to use, but has more limitations, with the limit of 20 MB for downloads and 50 MB for upload being the most impactful, and when we talk about sending and downloading files they are very important. On the other hand the API for clients has only a 2 GB per file limit.

With that in mind I decided to write some code in rust to do a proof-of-concept.

Connecting to Telegram

The first and most important step is to figure out how to connect to Telegram as a client, and for that we can use grammers, a very powerfull and simple to use library to interact with the Telegram API (both for bots and clients). Now that we have everything we can start writing some code.

Receiving the OTP code

Since we are using the client's API, we will need to authenticate with a phone number and then confirm with a code. For that let's create first a simple function that prompts a message to the user:

use std::io::{self, BufRead, Write};

fn prompt(message: &str) -> io::Result<String> {
    let mut stdout = io::stdout().lock();

    stdout.write_all(message.as_bytes())?;
    stdout.flush()?;

    let mut stdin = io::stdin().lock();
    let mut line = String::new();

    stdin.read_line(&mut line)?;

    Ok(line)
}

From grammers examples/downloader.rs

Ok done, now we can start the login step:

// Dependencies
use grammers_client::{Client, Config, SignInError};
use grammers_session::Session;

To persist the session, we can store it in a file to retrieve later:

const SESSION_FILE: &str = "/path/to/telegram.session";

For everything to work, you will need to use a async runtime. At the time I was using actix but you could replace it with regular a regular #[tokio::main]

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    // ...
}

Now we can initialize our client:

let telegram_client = Client::connect(Config {
    session: Session::load_file_or_create(SESSION_FILE)?,
    api_id: env::var("API_ID")
        .expect("API_ID not set")
        .parse()
        .expect("Failed to parse API_ID"),
    api_hash: env::var("API_HASH")
        .expect("API_HASH not set")
        .parse()
        .expect("Failed to parse API_HASH"),
    params: Default::default(),
}).await.unwrap();

And then sign in:

// Since we are using a file to persist the session, there is no need to login twice right?
if !telegram_client.is_authorized().await.unwrap() {
    println!("Signing in");
    let phone_number = prompt("Enter your phone number: ")?;
    let token = telegram_client
        .request_login_code(&phone_number)
        .await
        .unwrap();
    let code = prompt("Enter the code that you received: ")?;
    let signed_in = telegram_client.sign_in(&token, &code).await;

    match signed_in {
        Err(SignInError::PasswordRequired(pass_token)) => {
            let hint = pass_token.hint().unwrap();
            let prompt_msg = format!("Enter your password (hint {}): ", &hint);
            let password = prompt(prompt_msg.as_str())?;

            telegram_client
                .check_password(pass_token, password.trim())
                .await
                .unwrap();
        }
        Ok(_) => (),
        Err(err) => panic!("{}", err),
    }
}

Encrypting files

Another part of the idea is end-to-end encryption, and for this I will use orion to create some helper functions that will be used later:

use orion::aead;
use sha2::{Digest, Sha256};
use std::env;

fn encrypt(data: &[u8]) -> Vec<u8> {
    let mut hasher = Sha256::new();
    hasher.update(
        env::var("ENCRYPTION_KEY")
            .expect("Missing environment variable ENCRYPTION_KEY")
            .as_bytes(),
    );
    let pass_hash = hasher.finalize();
    let key = aead::SecretKey::from_slice(&pass_hash).unwrap();

    aead::seal(&key, data).unwrap()
}

fn decrypt(data: &[u8]) -> Vec<u8> {
    let mut hasher = Sha256::new();
    hasher.update(
        env::var("ENCRYPTION_KEY")
            .expect("Missing environment variable ENCRYPTION_KEY")
            .as_bytes(),
    );
    let pass_hash = hasher.finalize();
    let key = aead::SecretKey::from_slice(&pass_hash).unwrap();

    aead::open(&key, data).unwrap()
}

Is there a better way to handle this? Probably, but for now I will use it this way.

Sending messages

Now that we are connected and have the functions to encrypt, its time to actually send some messages:

// dependencies
use grammers_client::{
    client::auth::InvocationError,
    types::{Downloadable, Message},
    Client, InputMessage,
};
use grammers_session::{PackedChat, PackedType};
use std::{env, sync::Arc};
use tokio::sync::Mutex;

// This is only required when using it with actix
type TelegramClient = Arc<Mutex<Client>>;

To send a message, we need to specify a PackedChat struct for the send_message function. As it is very annoying to have to type everything over and over again, let's create a helper function to return the correct struct:

fn message_packet() -> PackedChat {
    PackedChat {
        ty: PackedType::Chat,
        id: env::var("GROUP_ID")
            .expect("Missing environment variable GROUP_ID")
            .parse()
            .expect("Failed to parse GROUP_ID to i64"),
        access_hash: None,
    }
}

Now let's create a function to send an empty message with a file:

async fn send_message_with_document(
    client: TelegramClient,
    filepath: &String,
) -> Result<Message, InvocationError> {
    let client = client.lock().await;
    let uploaded_file = (*client).upload_file(filepath).await.unwrap();
    (*client)
        .send_message(
            message_packet(),
            InputMessage::text("").document(uploaded_file),
        )
        .await
}

And to download the file later it is very simple:

async fn download_message_document(
    client: TelegramClient,
    message_id: i32,
    output_path: String,
) {
    let client = client.lock().await;
    let message = &(*client)
        .get_messages_by_id(message_packet(), &[message_id])
        .await
        .unwrap()[0];
    // No need to loop
    // There is only one message with the id

    match message {
        Some(msg) => {
            if let Some(media) = msg.media() {
                (*client)
                    .download_media(&Downloadable::Media(media), output_path)
                    .await
                    .expect("Failed to download file")
            }
        }
        None => {}
    }
}

Splitting the files into chunks

When the file is bigger than our upload limit or for faster upload/downloads, we will need to split the file into smaller chunks. For that we just need to open the file, split the bytes and then upload the chunks, simple enough right?

Not quite, we are talking of larger files over 2 GB. With that approach the file would need to be loaded in RAM first that could cause OOM to kill the process. And to solve this we will use mmap and memory mapped files, allowing us to read files without loading them entire into RAM, at the cost of more disk usage.

use memmap::Mmap;
use std::{fs, path::PathBuf};

// ~ 200 MB
const TARGET_CHUNK_SIZE: i64 = 209715200;
// 2GB
const TELEGRAM_MAX_FILESIZE: u64 = 2000000000;

async fn parse(...) {
    let file = fs::File::open(&filepath).unwrap();
    let filesize = file.metadata().unwrap().len();

    // If the file is smaller than 2 GB
    // we can just encrypt and upload
    if filesize < TELEGRAM_MAX_FILESIZE {
        let mmap_file = unsafe { Mmap::map(&file).unwrap() };
        let encrypted = encryption::encrypt(&mmap_file);

        // Send and store metadata about the file
        // in a database
        // ...
    } else {
        // If the file is bigger than 2GB
        // we will need to split the file into smaller chunks.
        let mmap_file = unsafe { Mmap::map(&file).unwrap() };
        let filesize = filesize as i64;
        // Calculate the amount of chunks based on the
        // target chunk size.
        // Smaller TARGET_CHUNK_SIZE equals more chunks
        let num_chunks = (filesize + TARGET_CHUNK_SIZE - 1) / TARGET_CHUNK_SIZE;
        let chunksize = (filesize + num_chunks - 1) / num_chunks;
        // Split the contents in chunks
        let chunks = mmap_file.chunks(chunksize as usize).collect::<Vec<&[u8]>>();

        for (index, chunk) in chunks.iter().enumerate() {
            let encrypted = encryption::encrypt(chunk);

            // Again, send and store metadata about the file
        }
    }
}

To assemble the file later you just need to do the opposite: download the chunks, loop over them and write the content (in order) into a single file.

Conclusion

This post is just a very basic showcase of the concept of a solution that is not perfect, download and upload speeds are slow, mostly because of Telegram being slow, and (again) there is also the issue of trust. There is also already a good project if you are looking for storing files on Telegram called Teldrive.

Overall, this is just a experiment that I made to try some new things and maybe learn something in the process.

See ya!

Cloudyy - Blog

Pi-Hole instance on the cloud & security

Backstory

Meet (.*?)

IP addresses change

Engineers solve practical problems

Storing some configs

Authentication with the API

Group IDs

What is my IP address

Whitelisting the client

What if I'm already blocked?

Conclusion

Using Telegram as infinite cloud storage

How that works?

Connecting to Telegram

Receiving the OTP code

Encrypting files

Sending messages

Splitting the files into chunks

Conclusion

Meet `(.*?)`