★ Transform your URLs with AI using our new Laravel package

I’m excited to share that we’ve released a new package called spatie/laravel-url-ai-transformer. This one can get the content of a webpage and transform it into something else using AI.

In this blog post, I’d like to share why I’ve created it and how you can use it.

The problem we wanted to solve

At Oh Dear, we’ve recently added a major new feature: we can now monitor servers via ping and monitor TCP ports.

As part of this change, we also added a couple of beautiful new feature pages, and took care of polishing the existing pages. We also wanted to include ld+json snippets to improve our SEO and make it easier for AIs to understand our website.

We could have just written one ld+json snippet and included that everywhere on our site. But we wanted to do better. Wouldn’t it be nice to have specific ld+json data on every page? And instead of having to write it manually, we could generate it with AI and periodically regenerate it so it always stays up to date.

Instead of hard coding all the logic into our app itself, I decided to extract all this to a package, so I can reuse it on other apps (and you can use it too).

How it works

Using our new spatie/laravel-url-ai-transformer package we will send the content of given URLs to the AI to transform. The transformation results will be stored in the database.

Under the hood, we use Prism for all AI interactions. This means you can easily switch between OpenAI, Anthropic Claude, Google Gemini, and other providers without changing your transformer code.

Let’s build a simple example that transforms blog posts into structured data. We’ll use the LdJsonTransformer that comes with the package to extract structured information from web pages.

First, you should use the Transform class to register URLs to transform, and which transformer to use:

use Spatie\LaravelUrlAiTransformer\Support\Transform;
use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;

// typically, in a service provider
Transform::urls(
    'https://spatie.be/blog',
    'https://spatie.be/open-source',
    'https://spatie.be/about-us'
)->usingTransformers(new LdJsonTransformer)

Here’s how that LdJsonTransformer class looks:

namespace Spatie\LaravelUrlAiTransformer\Transformers;

use Illuminate\Support\Str;
use Prism\Prism\Prism;
use Spatie\LaravelUrlAiTransformer\Support\Config;

class LdJsonTransformer extends Transformer
{
    public function transform(): void
    {
        $response = Prism::text()
            ->using(Config::aiProvider(), Config::aiModel())
            ->withPrompt($this->getPrompt())
            ->asText();

        $this->transformationResult->result = $response->text;
    }

    public function getPrompt(): string
    {
        return 'Summarize the following webpage to ld+json. Only return valid json, no backtick openings. Make the snippet as complete as possible. This is the content:'.Str::limit($this->urlContent, 6000);
    }
}

Of course, you can create any transformer you’d like, and for any purpose you like.

Now, you can transform the URLs by running:

php artisan transform-urls

This command will dispatch a queued job for each URL. Inside that queued job:

• the content of the URL will be fetched.
• the content will be sent to the configured AI.
• the response from the AI will be stored in the transformation_results table.

Here’s how you can retrieve transformation results:

use Spatie\LaravelUrlAiTransformer\Models\TransformationResult;

// Get structured data for a specific URL
$ldJsonData = TransformationResult::forUrl('https://spatie.be/blog', 'ldJson');

The first parameter is the URL, the second parameter is the transformer type. By default, transformer type is the lowercased class name of the transformer without the Transformer suffix.

Scheduling transformations

To keep your transformations up to date, schedule the command to run periodically:

// In app/Console/Kernel.php
protected function schedule(Schedule $schedule)
{
    $schedule->command('transform-urls')
        ->daily()
        ->at('02:00');
}

Transformers can decide when they should run using the shouldRun() method. Want a monthly report that only generates once every 30 days?

class MonthlyReportTransformer extends Transformer
{
    public function shouldRun(): bool
    {
        if ($this->transformationResult->successfully_completed_at === null) {
            return true;
        }
        
        return $this->transformationResult->successfully_completed_at->diffInDays() > 30;
    }
}

Instead of manually registering URLs, you can use the spatie/crawler package to crawl (parts of) your website.

Here’s an example of how to crawl all URLs of your website and pass it to the Transform class. Notice that we pass a closure to the urls method. By using a closure, the crawling will only happen when we actually perform the transformation, and not on each request.

namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlObservers\CrawlObserver;
use Spatie\Crawler\CrawlProfiles\CrawlInternalUrls;
use Spatie\LaravelUrlAiTransformer\Support\Transform;
use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;

class AiTransformerServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        Transform::urls(
            fn() => $this->crawlAllUrls()
        )->usingTransformers(new LdJsonTransformer());
    }

    protected function crawlAllUrls(): array
    {
        $startUrl = url('/');

        $urls = [];

        Crawler::create()
            ->setCrawlObserver(new class($urls) extends CrawlObserver
            {
                public function __construct(private array &$urls) {}

                public function crawled($url, $response, $foundOnUrl = null, ?string $linkText = null): void
                {
                    $this->urls[] = (string) $url;
                }
            })
            ->setCrawlProfile(new CrawlInternalUrls($startUrl))
            ->startCrawling($startUrl);

        return array_unique($urls);
    }
}

In closing

Our package has some more options that aren’t covered in this post. Head over to the spatie/laravel-url-ai-transformer docs to learn more.

Of course, this isn’t the first package that we’ve built. On our website you can see a list of all packages we’ve released previously.

If you want to support us, do take a look at our paid products: Flare, Mailcoach, Oh Dear and Ray.