Proxy pictrs requests through Lemmy (fixes #371) #77

Merged
nutomic merged 4 commits from pictrs-proxy into main 1 year ago
nutomic commented 1 year ago
Owner

Not sure which thumbnail sizes or other features we are using, but everything seems to work fine in the frontend.

I might also add rate limit and auth check for uploads while I'm at it, but not sure how to make it send the jwt string.

Not sure which thumbnail sizes or other features we are using, but everything seems to work fine in the frontend. I might also add rate limit and auth check for uploads while I'm at it, but not sure how to make it send the jwt string.
Owner

I'll try this out too. The main thing is that the urls don't change, and that its still accessible to the front end, and back end.

I'll try this out too. The main thing is that the urls don't change, and that its still accessible to the front end, and back end.
Owner

Take a look at the lib.rs fetch_pictrs function. I think that might need to be checked too, cause its required for thumbnails.

edit: actually maybe its fine, the docker pictrs didn't change.

Take a look at the `lib.rs fetch_pictrs` function. I think that might need to be checked too, cause its required for thumbnails. edit: actually maybe its fine, the docker pictrs didn't change.
Owner

The thumbnail fetching seems good.

One issue I did notice tho, is that I'm getting timeouts when I try to upload any file larger than ~1 MB.

lemmy_1     | [2020-08-04T15:31:31Z ERROR actix_http::response] Internal Server Error: Timeout
lemmy_1     | [2020-08-04T15:31:31Z INFO  actix_web::middleware::logger] 172.22.0.1:37376 "POST /pictrs/image HTTP/1.1" 500 34 "http://localhost:8536/create_post" "Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" 5.01821
The thumbnail fetching seems good. One issue I did notice tho, is that I'm getting timeouts when I try to upload any file larger than ~1 MB. ``` lemmy_1 | [2020-08-04T15:31:31Z ERROR actix_http::response] Internal Server Error: Timeout lemmy_1 | [2020-08-04T15:31:31Z INFO actix_web::middleware::logger] 172.22.0.1:37376 "POST /pictrs/image HTTP/1.1" 500 34 "http://localhost:8536/create_post" "Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0" 5.01821 ```
Poster
Owner

Okay I added a rate limit for image uploads, but it doesnt seem to work, I can still upload as many images in a row as I want...

I also tried to upload a 5.7 MB image (from this site) and it worked without problems in the docker/dev/ setup.

Okay I added a rate limit for image uploads, but it doesnt seem to work, I can still upload as many images in a row as I want... I also tried to upload a 5.7 MB image ([from this site](https://edmullen.net/test/imagefiletest.php)) and it worked without problems in the `docker/dev/` setup.
Owner

I think I know why image rate limiting isn't working, check the mod.rs impl RateLimit.

And then the image route needs image, not message.

      web::resource("/pictrs/image")
        .wrap(rate_limit.image())
        .route(web::post().to(upload)),
I think I know why image rate limiting isn't working, check the mod.rs `impl RateLimit`. And then the image route needs image, not message. ``` web::resource("/pictrs/image") .wrap(rate_limit.image()) .route(web::post().to(upload)), ```
Owner

Also I tried a few more pictures, and about 75% of them failed, with the same timeout error as above. It seems to stop working after the first few uploads go through.

Also I tried a few more pictures, and about 75% of them failed, with the same timeout error as above. It seems to stop working after the first few uploads go through.
Poster
Owner

Fixed the rate limit, but strangely I could still upload more than 20 images in a row, without any problems. So neither is the rate limit working, nor am I seeing the upload errors that you get.

Fixed the rate limit, but strangely I could still upload more than 20 images in a row, without any problems. So neither is the rate limit working, nor am I seeing the upload errors that you get.
Owner

Maybe its my local machine, I'll try deploying to test.lemmy.ml to see if it works.

Maybe its my local machine, I'll try deploying to test.lemmy.ml to see if it works.
Owner

Figured out the rate limit issue:

rate_limit/mod.rs, line 114 should be false

        RateLimitType::Image => {
          limiter.check_rate_limit_full(
            self.type_,
            &ip_addr,
            rate_limit.image,
            rate_limit.image_per_second,
            false,
          )?;

Otherwise its a check only.

Figured out the rate limit issue: `rate_limit/mod.rs`, line 114 should be false ``` RateLimitType::Image => { limiter.check_rate_limit_full( self.type_, &ip_addr, rate_limit.image, rate_limit.image_per_second, false, )?; ``` Otherwise its a check only.
Poster
Owner

Thanks, fixed. About your upload error, we could try how it goes running it on a server, or ask asonix if he has some idea.

Thanks, fixed. About your upload error, we could try how it goes running it on a server, or ask asonix if he has some idea.
nutomic merged commit c34fffc2c4 into main 1 year ago
Owner

I was wrong, I shouldn't have merged this. I'm getting that timeout error in production now.

I was wrong, I shouldn't have merged this. I'm getting that timeout error in production now.
The pull request has been merged as c34fffc2c4.
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This pull request currently doesn't have any dependencies.

Loading…
There is no content yet.