I’ve noticed that whenever I post from my instance (midwest.social) to lemmy.ml that the upvotes don’t match that of what’s on lemmy.ml and the comments don’t show up for a while.
You might want to change the log level for Lemmy to warn or error. I also checked the logs on our side just now, but there were no errors for midwest.social.
That one is probably harmless, it just means that the activity was already received and processed before. It is strange though that Lemmy on your instance would send the same activity twice.
Earlier I changed lemmy.ml logs to warn in docker-compose.yml and ran docker-compose up -d. Since then it has collected some better logs:
I suggest you follow the same steps for logging (with grep lemmy.ml), and see more useful log entries. In this case it looks like midwest.social is sometimes unreachable. In theory Lemmy will retry a few times, but that isnt really tested. And there are some weird public key errors, which should have more information in your logs.
sudo docker-compose logs -f --tail=100 lemmy
You might want to change the log level for Lemmy to warn or error. I also checked the logs on our side just now, but there were no errors for midwest.social.
lemmy_1 | 2022-01-12T21:46:01.367000Z WARN lemmy_apub_lib::activity_queue: Send https://midwest.social/activities/announce/d268e800-2ed3-42ba-8014-634373b46a96 to https://lemmy.ml/inbox failed with status 400 Bad Request: Failed to insert activity into database: duplicate key value violates unique constraint “idx_activity_ap_id”
That one is probably harmless, it just means that the activity was already received and processed before. It is strange though that Lemmy on your instance would send the same activity twice.
Earlier I changed lemmy.ml logs to warn in docker-compose.yml and ran
docker-compose up -d. Since then it has collected some better logs:$ sudo docker-compose logs lemmy | grep midwest.social |2022-01-12T22:17:30.845533ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/c3c3ff96-a8c4-453d-96c1-e02a7e8dcc41tohttps://midwest.social/inboxfailedwithstatus400BadRequest:error:0909006C:PEMroutines:get_name:nostartline:crypto/pem/pem_lib.c:745:Expecting:PUBLICKEY |2022-01-12T22:18:53.522306ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/a1c956e6-9b5b-4df0-a245-cefae75a39fbtohttps://midwest.social/inboxfailedwithstatus400BadRequest:error:0909006C:PEMroutines:get_name:nostartline:crypto/pem/pem_lib.c:745:Expecting:PUBLICKEY |2022-01-12T22:23:41.824212ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/0850d562-72c7-4df7-aa70-b0a4817e7564tohttps://midwest.social/inboxfailedwithstatus400BadRequest:Headerisexpired |2022-01-12T22:27:16.411288ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/a98ef44b-49c2-447a-b385-f0c90d7b9287tohttps://midwest.social/inboxfailedwithstatus504GatewayTimeout:<html> |2022-01-12T22:28:21.648136ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/ed0aa97b-2b6a-4e81-9941-bd736b2178e1tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout: |2022-01-12T22:28:43.533383ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/136c948f-fdc7-4a4e-949d-cdbc6bb21e21tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout: |2022-01-12T22:28:45.404798ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/626542e4-23b8-4b8e-981f-b5b14a821ac0tohttps://midwest.social/inboxfailedwithstatus408RequestTimeout: |2022-01-12T22:29:09.170348ZWARNlemmy_apub_lib::activity_queue:Sendhttps://lemmy.ml/activities/announce/36b21997-8f4d-43e5-8d8b-9491bb9d6d2ftohttps://midwest.social/inboxfailedwithstatus400BadRequest:HeaderisexpiredI suggest you follow the same steps for logging (with grep lemmy.ml), and see more useful log entries. In this case it looks like midwest.social is sometimes unreachable. In theory Lemmy will retry a few times, but that isnt really tested. And there are some weird public key errors, which should have more information in your logs.
Attaching to lemmy_lemmy_1 lemmy_1 | 2022-01-12T21:14:42.074410Z INFO lemmy_websocket::handlers: 93.43.251.95 joined lemmy_1 | 2022-01-12T21:14:44.929992Z INFO lemmy_server::scheduled_tasks: Done. lemmy_1 | 2022-01-12T21:14:44.930232Z INFO lemmy_server::scheduled_tasks: Reindexing table concurrently comment_aggregates ... lemmy_1 | 2022-01-12T21:14:46.286495Z INFO lemmy_server::scheduled_tasks: Done. lemmy_1 | 2022-01-12T21:14:46.286744Z INFO lemmy_server::scheduled_tasks: Reindexing table concurrently community_aggregates ... lemmy_1 | 2022-01-12T21:14:46.305327Z INFO lemmy_server::scheduled_tasks: Done. lemmy_1 | 2022-01-12T21:14:46.306563Z INFO lemmy_server::scheduled_tasks: Clearing old activities... lemmy_1 | 2022-01-12T21:14:47.128085Z INFO lemmy_server::scheduled_tasks: Done. lemmy_1 | 2022-01-12T21:14:50.284418Z INFO lemmy_websocket::handlers: 143.59.248.133 joined lemmy_1 | 2022-01-12T21:14:56.247886Z INFO Websocket Request{trace_id=00000000000000000000000000000000}:perform{self=CreateCommentLike { comment_id: CommentId(26276), score: 1, auth: Sensitive }}:send:send_activity_in_community:send_lemmy_activity: lemmy_apub::activities: Sending activity https://midwest.social/activities/like/4a8855b6-3d46-4a8e-8be3-edff0d15001c lemmy_1 | 2022-01-12T21:14:56.253978Z INFO lemmy_apub_lib::activity_queue: Sending https://midwest.social/activities/like/4a8855b6-3d46-4a8e-8be3-edff0d15001c to https://lemmy.ml/inbox lemmy_1 | 2022-01-12T21:16:01.872202Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/leftsthetics","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/nour","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/100510","cc":["https://lemmygrad.ml/c/leftsthetics"],"type":"Like","id":"https://lemmygrad.ml/activities/like/3bfcb836-e1c0-488b-9411-784c1db45095"},"cc":["https://lemmygrad.ml/c/leftsthetics/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880"} lemmy_1 | 2022-01-12T21:16:01.882757Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880 lemmy_1 | 2022-01-12T21:16:01.899481Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=dc91242d-6bf2-4478-b518-a2a5ee6f5873}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/640ff365-07c2-4eed-8e13-8790178cb880 lemmy_1 | 2022-01-12T21:16:03.079024Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/97958","cc":["https://lemmygrad.ml/c/communism"],"type":"Dislike","id":"https://lemmygrad.ml/activities/dislike/323009e1-31f9-49db-8131-61545e615742"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b"} lemmy_1 | 2022-01-12T21:16:03.084235Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b lemmy_1 | 2022-01-12T21:16:03.090818Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity:verify:verify:verify_person_in_community: lemmy_apub_lib::object_id: Fetching remote object https://lemmygrad.ml/u/Rylasasin lemmy_1 | 2022-01-12T21:16:03.205027Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=5c276d65-900c-4cf1-9ec3-f21b7e374e94}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/c919f208-a23a-47ae-a393-cd3fa6cdb88b lemmy_1 | 2022-01-12T21:16:04.732109Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/comment/97934","cc":["https://lemmygrad.ml/c/communism"],"type":"Dislike","id":"https://lemmygrad.ml/activities/dislike/a50ea1fe-31c5-4536-833d-af91172eb2df"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a"} lemmy_1 | 2022-01-12T21:16:04.736151Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a lemmy_1 | 2022-01-12T21:16:04.743282Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=48268bff-7a4d-4fb4-8a6d-fd9ad5b36278}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/297a68c6-102d-46da-b0be-8174fe42c57a lemmy_1 | 2022-01-12T21:19:19.390390Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor":"https://lemmygrad.ml/c/communism","to":["https://www.w3.org/ns/activitystreams#Public"],"object":{"actor":"https://lemmygrad.ml/u/Rylasasin","to":["https://www.w3.org/ns/activitystreams#Public"],"object":"https://lemmygrad.ml/post/83464","cc":["https://lemmygrad.ml/c/communism"],"type":"Like","id":"https://lemmygrad.ml/activities/like/16204a7e-1883-49bd-9094-e755d6b4d911"},"cc":["https://lemmygrad.ml/c/communism/followers"],"type":"Announce","id":"https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d"} lemmy_1 | 2022-01-12T21:19:19.395819Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox:receive_activity: lemmy_apub::http: Verifying activity https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d lemmy_1 | 2022-01-12T21:19:19.404319Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=ca0ed8b2-2e7c-443f-a624-df2df6015b67}:shared_inbox:receive_activity: lemmy_apub::http: Receiving activity https://lemmygrad.ml/activities/announce/231157fc-98d9-49bd-bc8a-906e67076d6d lemmy_1 | 2022-01-12T21:19:20.534398Z INFO HTTP request{http.method=POST http.scheme="http" http.host=midwest.social http.target=/inbox otel.kind="server" request_id=b0fc1656-571e-4bd4-80f4-c2088934dab0}:shared_inbox: lemmy_apub::http: Received shared inbox activity {"@context":["https://www.w3.org/ns/activitystreams",{"stickied":"as:stickied","pt":"https://join-lemmy.org#","sc":"http://schema.org#","matrixUserId":{"type":"sc:Text","id":"as:alsoKnownAs"},"sensitive":"as:sensitive","comments_enabled":{"type":"sc:Boolean","id":"pt:commentsEnabled"},"moderators":"as:moderators"},"https://w3id.org/security/v1"],"actor