Notion API Pagination

2024-12-19 21:03 UTC gpt-4o Open in ChatGPT ↗

You

Notion api only gives me 100 results:

def get_db(url: str): return requests.post(url, headers=headers).json()[“results”]

i want to get all results

ChatGPT

The Notion API has a default page size limit of 100 results per request. To retrieve more than 100 results, you need to use pagination with the start_cursor parameter. Here’s how you can modify your get_db function to fetch all results:

import requests

def get_all_results(url: str):
    """
    Fetches all results from a Notion database by handling pagination.
    """
    all_results = []
    has_more = True
    start_cursor = None

    while has_more:
        payload = {"start_cursor": start_cursor} if start_cursor else {}
        response = requests.post(url, headers=headers, json=payload).json()
        results = response.get("results", [])
        all_results.extend(results)

        has_more = response.get("has_more", False)
        start_cursor = response.get("next_cursor")

    return all_results

Explanation:

1. has_more: This field in the API response indicates if there are more results to fetch.
2. start_cursor: The cursor for the next set of results, provided in the next_cursor field of the API response.
3. Loop Until All Results Are Fetched: The function continues to make requests until has_more is False.

Usage Example:

url = "https://api.notion.com/v1/databases/YOUR_DATABASE_ID/query"
all_results = get_all_results(url)
print(f"Total results fetched: {len(all_results)}")

This ensures that your function fetches all results, no matter how many there are.

You

I am getting the main page text of each notion page and turning into html text.

however bullet lists don’t seem to work:

Get page main text

# -------------------------------------------------------------------------
blocks_response = requests.get(f"https://api.notion.com/v1/blocks/{page_id}/children", headers=headers)
blocks = blocks_response.json()["results"]
page_text = [b[0]["text"]["content"] for b in [block["paragraph"]["rich_text"] for block in blocks] if len(b) > 0]

ChatGPT

The issue with bullet lists in Notion pages likely arises because you are treating every block as a paragraph block. Bullet lists in Notion are represented as bulleted_list_item blocks in the API. To handle bullet lists and convert them to HTML properly, you need to check the block type and process it accordingly.

Here’s an updated and enhanced version of your code that includes support for bullet lists:

import requests

def get_page_text_as_html(page_id: str):
    """
    Fetches the main text content of a Notion page and converts it to HTML.
    Handles paragraphs and bullet lists.
    """
    # Get page blocks
    blocks_response = requests.get(
        f"https://api.notion.com/v1/blocks/{page_id}/children", headers=headers
    )
    blocks = blocks_response.json().get("results", [])

    html_content = ""

    for block in blocks:
        block_type = block["type"]
        if block_type == "paragraph":
            # Extract text from a paragraph
            rich_text = block["paragraph"]["rich_text"]
            if rich_text:
                text_content = "".join([rt["text"]["content"] for rt in rich_text])
                html_content += f"<p>{text_content}</p>\n"

        elif block_type == "bulleted_list_item":
            # Extract text from a bullet list item
            rich_text = block["bulleted_list_item"]["rich_text"]
            if rich_text:
                text_content = "".join([rt["text"]["content"] for rt in rich_text])
                html_content += f"<li>{text_content}</li>\n"

    # Wrap bullet list items in a <ul> if they exist
    if "<li>" in html_content:
        html_content = html_content.replace("<li>", "<ul>\n<li>", 1)
        html_content = html_content.replace("</li>", "</li>\n</ul>", 1)

    return html_content

Key Points:

1. Block Type Handling:

   • The block type is determined by block["type"].
   • paragraph blocks are wrapped in <p> tags.
   • bulleted_list_item blocks are wrapped in <li> tags.

2. Rich Text Extraction:

   • The rich_text field contains the actual text content.
   • Use a list comprehension to extract and join text fragments.

3. Wrapping Bullet Lists:

   • If <li> tags are detected, ensure they are wrapped in a <ul> tag for proper HTML structure.

Example Usage:

page_id = "YOUR_PAGE_ID"
html_text = get_page_text_as_html(page_id)
print(html_text)

This function generates valid HTML text for paragraphs and bullet lists, ensuring bullet lists are correctly structured in <ul> tags. You can extend this function further to handle other block types, such as numbered lists, headings, and images, if needed.

You

also maintain any text decorations like bold and italic

ChatGPT

To maintain text decorations such as bold, italic, or underline, you need to inspect the annotations field of the rich_text objects. Here’s an updated version of your function that supports text decorations:

import requests

def get_page_text_as_html_with_decorations(page_id: str):
    """
    Fetches the main text content of a Notion page and converts it to HTML.
    Handles paragraphs, bullet lists, and text decorations.
    """
    # Get page blocks
    blocks_response = requests.get(
        f"https://api.notion.com/v1/blocks/{page_id}/children", headers=headers
    )
    blocks = blocks_response.json().get("results", [])

    html_content = ""

    def apply_decorations(text: str, annotations: dict) -> str:
        """
        Applies text decorations (bold, italic, underline) based on annotations.
        """
        if annotations.get("bold"):
            text = f"<b>{text}</b>"
        if annotations.get("italic"):
            text = f"<i>{text}</i>"
        if annotations.get("underline"):
            text = f"<u>{text}</u>"
        if annotations.get("strikethrough"):
            text = f"<s>{text}</s>"
        if annotations.get("code"):
            text = f"<code>{text}</code>"
        return text

    for block in blocks:
        block_type = block["type"]
        if block_type == "paragraph":
            # Extract and decorate text from a paragraph
            rich_text = block["paragraph"]["rich_text"]
            if rich_text:
                decorated_text = "".join(
                    [apply_decorations(rt["text"]["content"], rt["annotations"]) for rt in rich_text]
                )
                html_content += f"<p>{decorated_text}</p>\n"

        elif block_type == "bulleted_list_item":
            # Extract and decorate text from a bullet list item
            rich_text = block["bulleted_list_item"]["rich_text"]
            if rich_text:
                decorated_text = "".join(
                    [apply_decorations(rt["text"]["content"], rt["annotations"]) for rt in rich_text]
                )
                html_content += f"<li>{decorated_text}</li>\n"

    # Wrap bullet list items in a <ul> if they exist
    if "<li>" in html_content:
        html_content = html_content.replace("<li>", "<ul>\n<li>", 1)
        html_content = html_content.replace("</li>", "</li>\n</ul>", 1)

    return html_content

Enhancements:

1. apply_decorations Function:

   • This function checks the annotations field of each rich_text object and wraps the text in appropriate HTML tags.
   • Supports:
     • Bold: <b>
     • Italic: <i>
     • Underline: <u>
     • Strikethrough: <s>
     • Code: <code>

2. Text Decoration Application:

   • Each rich_text element’s text["content"] is processed with the apply_decorations function before being joined into the final string.

3. HTML Structure Maintenance:

   • Properly wraps bullet list items in <ul> tags.

Example Usage:

page_id = "YOUR_PAGE_ID"
html_text = get_page_text_as_html_with_decorations(page_id)
print(html_text)

Sample Output:

For a page with:

• A bold paragraph
• An italic bullet list item

The output will be:

<p><b>This is bold text</b></p>
<ul>
<li><i>This is italic bullet list</i></li>
</ul>

This ensures that the HTML representation closely mirrors the original Notion page, including its text decorations.

You

make it maintain url hrefs as well

ChatGPT

To maintain URLs (hrefs) in the HTML output, you need to inspect the href field of the rich_text objects. If href is present, wrap the text in an <a> tag with the corresponding href.

Here’s an updated version of the function that includes support for URLs:

import requests

def get_page_text_as_html_with_decorations_and_links(page_id: str):
    """
    Fetches the main text content of a Notion page and converts it to HTML.
    Handles paragraphs, bullet lists, text decorations, and links.
    """
    # Get page blocks
    blocks_response = requests.get(
        f"https://api.notion.com/v1/blocks/{page_id}/children", headers=headers
    )
    blocks = blocks_response.json().get("results", [])

    html_content = ""

    def apply_decorations(text: str, annotations: dict, href: str = None) -> str:
        """
        Applies text decorations (bold, italic, underline) and wraps links based on annotations.
        """
        if annotations.get("bold"):
            text = f"<b>{text}</b>"
        if annotations.get("italic"):
            text = f"<i>{text}</i>"
        if annotations.get("underline"):
            text = f"<u>{text}</u>"
        if annotations.get("strikethrough"):
            text = f"<s>{text}</s>"
        if annotations.get("code"):
            text = f"<code>{text}</code>"
        if href:
            text = f"<a href='{href}' target='_blank'>{text}</a>"
        return text

    for block in blocks:
        block_type = block["type"]
        if block_type == "paragraph":
            # Extract and decorate text from a paragraph
            rich_text = block["paragraph"]["rich_text"]
            if rich_text:
                decorated_text = "".join(
                    [apply_decorations(rt["text"]["content"], rt["annotations"], rt["href"]) for rt in rich_text]
                )
                html_content += f"<p>{decorated_text}</p>\n"

        elif block_type == "bulleted_list_item":
            # Extract and decorate text from a bullet list item
            rich_text = block["bulleted_list_item"]["rich_text"]
            if rich_text:
                decorated_text = "".join(
                    [apply_decorations(rt["text"]["content"], rt["annotations"], rt["href"]) for rt in rich_text]
                )
                html_content += f"<li>{decorated_text}</li>\n"

    # Wrap bullet list items in a <ul> if they exist
    if "<li>" in html_content:
        html_content = html_content.replace("<li>", "<ul>\n<li>", 1)
        html_content = html_content.replace("</li>", "</li>\n</ul>", 1)

    return html_content

Key Changes:

1. href Handling:

   • The href field is checked for each rich_text object.
   • If href is present, the text is wrapped in an <a> tag with the href attribute.
   • Added target="_blank" to open links in a new tab.

2. Enhanced apply_decorations Function:

   • Accepts href as a parameter and includes it in the <a> tag if present.
   • Combines decorations with links seamlessly.

Example Usage:

page_id = "YOUR_PAGE_ID"
html_text = get_page_text_as_html_with_decorations_and_links(page_id)
print(html_text)

Sample Output:

For a Notion page with:

• A bold paragraph containing a link
• A bullet list item with an underlined link

The output will be:

<p><b>This is bold text with a <a href='https://example.com' target='_blank'>link</a></b></p>
<ul>
<li><u><a href='https://anotherlink.com' target='_blank'>Underlined link</a></u></li>
</ul>

This ensures that both text decorations and links (with their hrefs) are properly represented in the HTML output.