Newbie question

[snipecoder]

Hello

I can see:

2024-05-26T18:34:07.967Z [NOTICE] [-] [mongodb] --------------------------------------

2024-05-26T18:34:07.967Z [NOTICE] [-] [mongodb] Successfully stored the delivery log for message_id : 633deaee03f57777a494bd0a71319f95@example.com !!!

2024-05-26T18:34:07.967Z [NOTICE] [-] [mongodb] --------------------------------------

But I don't see the email delivery object, with the subject, mail from etc, should i enable this in the config?

[thenitai]

Go to your favorite mongodb client and you should see a collection called "email_delivery_results" (you can set the name of the collection in the mongodb.ini file). Query the collection and you will see the documents :)

Hope this helps.

[snipecoder]

Yes, I have checked, that is all I see:

On the other hand, I don't see email_incoming_haraka collection:

Config:

; This file must be placed in "config" directory of your Haraka server.

;
; MongoDB Credentials
;

[mongodb]
user=admin
pass=removed
host=x.x.x.x
port=27017
db=outbound
string=mongodb://admin:removed@x.x.x.x:27017/?authSource=admin
restart=yes

; collection name
[collections]
queue=email_incoming_haraka
delivery=email_delivery_results

; Absolute path to store attachments
[attachments]
path=/opt/attachments
; The below path has to exists or else we stop Haraka from starting. Leave empty if you don't want to execute a check.
; Great if you have to make sure that a value is mounted or keeps being mounted. If we loose connection we exit Haraka.
path_check=
; Attachments that should always be rejected. The values below are the ones that Gmail and Outlook also reject
reject=['application/x-freearc','application/x-bzip','application/x-bzip2','application/x-csh','application/gzip','application/java-archive','text/javascript','application/vnd.apple.installer+xml','application/x-httpd-php','application/x-sh','application/xhtml+xml','application/vnd.microsoft.portable-executable','application/x-msdownload','application/exe','application/x-exe','application/dos-exe','vms/exe','application/x-winexe','application/msdos-windows','application/x-msdos-program']
custom_content_type={ 'application/imed' : ['imed'] }
; You can also reject attachments by filename. Enter the complete filename or only part of it
reject_by_name=['~WRD']

; Convert inline images or not
; cid = leave value as cid:(number) - this is useful if you want to process the images later on
; base64 = convert inline images to base64 - will convert inline images to base64 and remove it from the attachment array
; path = convert the cid:(number) value to the path given, the filename will be appended to your path, e.g., "(path)/image.png".
; Do NOT append a "/" at the end of your path
cid = cid

; Enable the section you want
[enable]
queue=yes
delivery=yes

; Message settings
; MongoDB has a limit of 16 MB per document. Hence you might want to check the size of the message (https://docs.mongodb.com/manual/reference/limits/)
[message]
limit=16777216
; Store the whole raw email and the parsed haraka body (Enabling this will create large documents and should only be used for debugging)
store_raw=no

; SMTP settings for error messages
; Define your custom smtp server settings to send message when a message is to large or there is an error with inserting the message
[smtp]
user=removed
pass=removed
host=removed
port=587
tls=yes
ssl=no
msg_limit=Your message could not be accepted as the message body is over 16 MB. Please remove parts of the email and send again.
msg_error_insert=An error occurred accepting your message. Please send again.
msg_error_parsing=We are not able to parse this message properly. Please make sure this email is RFC body conform.
from=mailer-daemon@relay.mailwish.com
; CC and BCC can be a comma separated list of email addresses
cc=
bcc=

; Limit settings
; Limit incoming messages. Please see the README for instructions
[limits]
incoming=no

; Limit in seconds
incoming_seconds=30

; Strings that should be excluded from the limits check
exclude[] = helpmonks
exclude[] = sendmonks
exclude[] = razuna

; String that should be included in the limits check
include[] = monitor
include[] = notification
include[] = java
include[] = noreply
include[] = notreply
include[] = no-reply
include[] = not-reply
include[] = deployment
include[] = notifier
include[] = root
include[] = alert
include[] = uptime
include[] = robot
include[] = opsgenie

; Set the database to be used. Either "mongodb" or "redis"
db=redis

; Set the collection to be used if you use mongodb
incoming_collection=limit_incoming

; Redis Credentials
; Used for the limit incoming option above
[redis]
; user
user=
; password
pass=
; host
host=127.0.0.1
; string (full redis connection string)
string=
; port
port=6379

Last question, will I see the message content? sometimes I need the message content to check if not spam.

[thenitai]

The delivery looks correct. There are three hooks: delivery, send, mx. Each one is a separate document, but they all contain the same message_id. This is what Haraka returns.

Regarding incoming, once you have Haraka set up and MongoDB added to the plugin section, you should see the emails in the incoming collection. Everything is stored.

[snipecoder]

Per the code, I see that more data is stored like Subject, sender, and recipient, where can I see this?

[snipecoder]

Here is the part of the code I saw, that is not stored:

// Mail object
		var _email = {
			'haraka_body': _store_raw && body ? body : {},
			'raw_html': _body_html,
			'raw_text': _body_text,
			'raw': _store_raw ? email_object : {},
			'from': email_object.headers.get('from') ? email_object.headers.get('from').value : null,
			'to': email_object.headers.get('to') ? email_object.headers.get('to').value : null,
			'cc': email_object.headers.get('cc') ? email_object.headers.get('cc').value : null,
			'bcc': email_object.headers.get('bcc') ? email_object.headers.get('bcc').value : null,
			'subject': email_object.subject,
			'date': email_object.date || email_object.headers.get('date'),
			'received_date': _now,
			'message_id': email_object.messageId ? email_object.messageId.replace(/<|>/gm, '') : new ObjectID() + '@haraka-helpmonks.com',
			'attachments': email_object.attachments || [],
			'headers': email_object.headers,
			'html': email_object.html,
			'text': email_object.text ? email_object.text : null,
			'timestamp': _now,
			'status': 'unprocessed',
			'source': 'haraka',
			'in_reply_to' : email_object.inReplyTo,
			'reply_to' : email_object.headers.get('reply-to') ? email_object.headers.get('reply-to').value : null,
			'references' : email_object.references,
			'pickup_date' : _now,
			'mail_from' : connection && connection.transaction ? connection.transaction.mail_from : null,
			'rcpt_to' : connection && connection.transaction ? connection.transaction.rcpt_to : null,
			'size' : connection && connection.transaction ? connection.transaction.data_bytes : null,
			'transferred' : false,
			'processed' : false,
			'extracted_html_from': email_object.extracted_html_from,
			'extracted_text_from': email_object.extracted_text_from
		};

In the code, I also saw this:

plugin.lognotice('--------------------------------------');
						plugin.lognotice(` Successfully stored the email with the message_id: ${_email.message_id} !!! `);
						plugin.lognotice('--------------------------------------');

I never saw the above output, I see only this:

2024-05-28T06:45:14.501Z [NOTICE] [-] [mongodb] --------------------------------------

2024-05-28T06:45:14.501Z [NOTICE] [-] [mongodb] Successfully stored the delivery log for message_id : 51a6343ce719242c0ea47028c3373944@mindfultrade.online !!!

2024-05-28T06:45:14.501Z [NOTICE] [-] [mongodb] --------------------------------------

I don't rcpt, sender, subject etc.

[thenitai]

As mentioned, delivery results are based on what Haraka provides. All fields are stored in incoming emails.

[snipecoder]

I don't see that fields... subject etc are not there.
When I used another plugin that was there:
https://github.com/mr-karan/haraka-plugin-outbound-logger

[snipecoder]

Can you tell me where that part is saved and why is in the code?

// Mail object
		var _email = {
			'haraka_body': _store_raw && body ? body : {},
			'raw_html': _body_html,
			'raw_text': _body_text,
			'raw': _store_raw ? email_object : {},
			'from': email_object.headers.get('from') ? email_object.headers.get('from').value : null,
			'to': email_object.headers.get('to') ? email_object.headers.get('to').value : null,
			'cc': email_object.headers.get('cc') ? email_object.headers.get('cc').value : null,
			'bcc': email_object.headers.get('bcc') ? email_object.headers.get('bcc').value : null,
			'subject': email_object.subject,
			'date': email_object.date || email_object.headers.get('date'),
			'received_date': _now,
			'message_id': email_object.messageId ? email_object.messageId.replace(/<|>/gm, '') : new ObjectID() + '@haraka-helpmonks.com',
			'attachments': email_object.attachments || [],
			'headers': email_object.headers,
			'html': email_object.html,
			'text': email_object.text ? email_object.text : null,
			'timestamp': _now,
			'status': 'unprocessed',
			'source': 'haraka',
			'in_reply_to' : email_object.inReplyTo,
			'reply_to' : email_object.headers.get('reply-to') ? email_object.headers.get('reply-to').value : null,
			'references' : email_object.references,
			'pickup_date' : _now,
			'mail_from' : connection && connection.transaction ? connection.transaction.mail_from : null,
			'rcpt_to' : connection && connection.transaction ? connection.transaction.rcpt_to : null,
			'size' : connection && connection.transaction ? connection.transaction.data_bytes : null,
			'transferred' : false,
			'processed' : false,
			'extracted_html_from': email_object.extracted_html_from,
			'extracted_text_from': email_object.extracted_text_from
		};

[snipecoder]

This function does not store, and I need this function badly:
queue_to_mongodb

Any idea why?

[thenitai]

This function does not store, and I need this function badly: queue_to_mongodb

Any idea why?

No, I don't know. We use the very same plugin with a million emails a day, so I can say with certainty that it works :)

[snipecoder]

Nothing seems to go smoothly for me, not even getting fired for that function.
Very weird.

[snipecoder]

I found the issue, in your code you call queue:
plugin.register_hook('queue', 'queue_to_mongodb');

Seems like this wrong, how that working for you??
I change to:
plugin.register_hook('queue_outbound', 'queue_to_mongodb');

And now working, am I missing something?
https://haraka.github.io/core/Outbound#the-queue_outbound-hook

I want to ask if you can store a message (in message/rfc822 format)?

I saw this message:

2024-05-28T17:21:12.277Z [NOTICE] [-] [mongodb] body_info.meta !!! has_rfc_822_message=false is_html_from_text=false html_source=mailparser_html html_has_valid_encoding=false text_source=mailparser_text text_has_valid_encoding=false does_bodytext_contain_invalid_html="" does_body_text_encoded_contain_invalid_html="" does_bodytext_contain_replacement_char_unicode="" does_body_text_encoded_contain_replacement_char_unicode=""