Créer un téléchargeur Google Drive dans Golang (partie 1)

Introduction

In this tutorial, we’ll build a powerful downloader that allows downloading files from Google Drive and other cloud providers. With Golang’s efficient concurrency patterns, you'll be able to manage multiple downloads concurrently, stream large files, and track progress in real-time. Whether you’re downloading a few small files or handling large data sets, this project will showcase how to build a scalable and robust downloader that can easily be extended to support multiple cloud platforms.

If you're looking for a way to simplify and automate downloading large files, this tutorial is perfect for you. By the end, you’ll have a flexible and customizable Go-based downloader to suit your needs.

In a hurry?

If you're just looking to use this downloader with a UI, visit evolveasdev.com for read the full article & Go Downloader's Github. You'll find the docs to get it running fast.

What You’ll Learn

• Go Concurrency Patterns:
  Learn how to use Goroutines, channels and mutexes to handle multiple concurrent file downloads efficiently.

• Streaming Large Downloads:
  Explore how to stream large files while managing memory and system resources effectively.

• Concurrent File Downloads:
  Understand how to download files concurrently, speeding up the process and improving performance.

• Real-Time Progress Updates:
  Implement progress tracking to provide real-time feedback on download status.

• Handling Interruptions and Cancellations:
  Learn how to gracefully cancel one or all ongoing downloads.

Note: This tutorial will only focus on the core downloading logic.

Environment Setup

First before doing anything make sure to properly setup your environment to avoid potential bugs in future.

Prerequisites

• Install Go
• AIR for auto reloading
• Makefile to run complex commands
• Goose for PostgreSQL migrations

Configuring Makefile

Create a makefile at the root of the project with the following.

# Load environment variables from .env file
include ./.env

# To run the application
run: build
    @./bin/go-downloader

# Build the application
build:
    @go build -tags '!dev' -o bin/go-downloader

# Database migration status
db-status:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) status

# Run database migrations
up:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) up

# Roll back the last database migration
down:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) down

# Reset database migrations
reset:
    @GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) reset

High-Level Folder Structure Overview

go-downloader/
├── api
├── config
├── migrations
├── service
├── setting
├── store
├── types
├── util
├── .env
├── .air.toml
├── Makefile
├── go.mod
├── go.sum
└── main.go

Setting environment variables

Create a .env file in root or handle environment variables however you like, we'll use joho/godotenv package.

GOOGLE_CLIENT_ID
GOOGLE_CLIENT_SECRET
SESSION_SECRET=something-super-secret
APP_URL=http://localhost:3000
POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_DB

Creating a Web Server

We'll now start creating the web server that'll handle all the incoming requests.

Heads Up! The main part of this guide begins here. Get ready to dive in!

API Layer

To start, create the following files inside api folder api.go and route.go

route.go file setup

All API routes will be defined in this here. We create a NewRouter struct that takes the env configuration, allowing all routes and handlers to access environment variables.

package api

import (
    "github.com/gofiber/fiber/v2"
    "github.com/nilotpaul/go-downloader/config"
)

type Router struct {
    env       config.EnvConfig
}

func NewRouter(env config.EnvConfig) *Router {
    return &Router{
        env:      env,
    }
}

func (h *Router) RegisterRoutes(r fiber.Router) {
    r.Get("/healthcheck", func(c *fiber.Ctx) error {
        return c.JSON("OK")
    })
}

api.go file setup

Here, we’ll add all the necessary middlewares, such as CORS and logging, before starting the server.

type APIServer struct {
    listenAddr string
    env        config.EnvConfig
}

func NewAPIServer(listenAddr string, env config.EnvConfig) *APIServer {
    return &APIServer{
        listenAddr: listenAddr,
        env:        env,
    }
}

func (s *APIServer) Start() error {
    app := fiber.New(fiber.Config{
        AppName:      "Go Downloader",
    })

    handler := NewRouter()
    handler.RegisterRoutes(app)

    log.Printf("Server started on http://localhost:%s", s.listenAddr)

    return app.Listen(":" + s.listenAddr)
}

Main Entrypoint

This is the main package in main.go file which will act as a entrypoint to the whole.

func main() {
    // Loads all Env vars from .env file.
    env := config.MustLoadEnv()

    log.Fatal(s.Start())
}

This is enough to start up the server and test it.

Start the server

air

that's it.?

Testing

curl http://localhost:3000/healthcheck

The response should be OK with status 200

Creating a Provider Store

We need to implement a scalable solution for adding support for multiple cloud providers if necessary.

Working on provider registry

// Better to keep it in a seperate folder.
// Specific only to OAuth Providers.
type OAuthProvider interface {
    Authenticate(string) error
    GetAccessToken() string
    GetRefreshToken() string
    RefreshToken(*fiber.Ctx, string, bool) (*oauth2.Token, error)
    IsTokenValid() bool
    GetAuthURL(state string) string
    CreateOrUpdateAccount() (string, error)
    CreateSession(c *fiber.Ctx, userID string) error
    UpdateTokens(*GoogleAccount) error
}

type ProviderRegistry struct {
    Providers map[string]OAuthProvider
}

func NewProviderRegistry() *ProviderRegistry {
    return &ProviderRegistry{
        Providers: make(map[string]OAuthProvider),
    }
}

func (r *ProviderRegistry) Register(providerName string, p OAuthProvider) {
    r.Providers[providerName] = p
}

func (r *ProviderRegistry) GetProvider(providerName string) (OAuthProvider, error) {
    p, exists := r.Providers[providerName]
    if !exists {
        return nil, fmt.Errorf("Provider not found")
    }

    return p, nil
}

The ProviderRegistry serves as a central map to hold all our OAuth providers. When we initialize our providers, we'll register them in this map. This allows us to easily access any registered provider's functionalities throughout our service.

You'll see this action later.

Initializing the Provider Store

We'll register our providers based on the environment variables provided.

func InitStore(env config.EnvConfig) *ProviderRegistry {
    r := NewProviderRegistry()

    if len(env.GoogleClientSecret) != 0 || len(env.GoogleClientID) != 0 {
        googleProvider := NewGoogleProvider(googleProviderConfig{
            googleClientID:     env.GoogleClientID,
            googleClientSecret: env.GoogleClientSecret,
            googleRedirectURL:  env.AppURL + "/callback/google",
        }, env)

        r.Register("google", googleProvider)
    }

    return r
}

Read the full article here.

Conclusion

Nous avons jeté les bases du Google Drive Downloader dans Go, en couvrant des éléments clés tels que la configuration de la structure du projet, la gestion de Google OAuth et la pose des bases d'une expansion future. En chemin, nous avons abordé quelques sujets importants :

• Migrations de bases de données avec Goose pour assurer une gestion des données fluide et évolutive.
• Création d'un registre de fournisseurs pour étendre facilement la prise en charge de fournisseurs de stockage cloud supplémentaires à l'avenir.
• Concevoir une architecture flexible qui facilite la gestion ultérieure d'une logique plus complexe.

C'est largement suffisant pour un seul post, car les choses devenaient assez longues ! Nous reviendrons dans la Partie 2 pour terminer notre travail, où nous travaillerons sur la principale fonctionnalité de téléchargement.

En attendant, n'hésitez pas à explorer l'implémentation actuelle dans mon GitHub et restez à l'écoute pour les prochaines étapes. Bon téléchargement !

Ce qui précède est le contenu détaillé de. pour plus d'informations, suivez d'autres articles connexes sur le site Web de PHP en chinois!